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Abstract. 

“The  purpose  of  the  Advanced  Decision  Architectures  (ADA)  Collaborative  Technology  Alliance  (CTA)  was  to 
design,  test,  and  transition  new  cognitive  and  computer  science  innovations  to  facilitate  better  Soldier  understanding 
of  the  tactical  situation,  more  thorough  evaluation  of  courses  of  action,  and,  ultimately,  better  and  more  timely 
decisions.”2  The  ADA  CTA  was  conducted  from  2001  to  2009  with  final  reports  being  published  in  2010.  It  had  4 
research  area  technical  objectives: 

“Cognitive  Modeling  and  Metrics 

Advance  human  decision-making  technologies  through  research  on  cognitive  processing,  modeling,  and  metrics  to 
serve  as  a  basis  for  describing  individual  and  team  accomplishment  of  the  tasks  to  be  performed. 

Team  Communication  and  Collaboration 

Develop  analytical  tools  and  techniques  to  increase  shared  situational  awareness  for  collaborative  decision-making 
in  real-time,  distributed,  mobile  environments. 

Context-Sensitive  Information  Presentation 

Reduce  physical  and  cognitive  complexity  of  Soldier-system  interaction  by  developing  approaches  for  integrating 
information  over  diverse  sources  and  perspectives,  multiple  modalities,  and  for  mixed  human-system/agent  teams. 

Fusion  and  Intelligent  Architectures 

Support  both  end-user  applications  and  the  computational  and  communication  infrastructure  via  developing 
intelligent  architectures  for  fusion  and  planning  and  agile  computing  approaches  for  decision  support.” 

This  annotated  bibliography  documents  the  ADA  CTA  body  of  work  and  provides  glimpses,  through  report 
abstracts,  of  the  work  performed  in  pursuit  of  each  of  these  technical  objectives. 


2  http://www.arl. army, mil/www/default.cfm?Action=93&Page=153 
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1.  Introduction. 

This  purpose  of  this  annotated  bibliography  is  to  document  the  research  and  resulting  reports  and  publications 
produced  by  the  Advanced  Decision  Architectures  (ADA)  consortium  of  the  Collaborative  Technology  Alliance 
(CTA)  between  the  U.S.  Army  Research  Laboratory  and  industry  and  academia  between  2001  and  the  conclustion 
of  the  CTA  in  2010. 

1.1.  Performing  Organizations. 

The  studies  documented  in  this  report  were  performed  by  the  following  organizations: 


<2> 

A  L  I  O  N 

SCIENCE  AND  TECHNOLOGY 


y  klein 

ASSOCIATES 

A  Division  of  Applied  Research  Associates,  Inc. 


^  0: 

* 

T  -  H  -  E 

OHIO 

STATE 

UNIVERSITY 

Mlir 

Massachusetts 

Institute  of 
Technology 

Y  A  University  of  i 

West  Florida  1 

Carnegie  Mellon 


gS'”  University  of 

Central 

Florida 


UNIVERSITY  OF 

MARYLAND 


WRIGHT  STATE 


^JxNIVERSITY  OF  MICHIGAN 


Figure  1  -  Organizations  Participating  in  the  Collaborative  Technology  Alliance1 


1.2.  Research  Objectives  of  the  ADA  CTA. 

The  study  goals  of  the  ADA  CTA  were  divided  into  4  Task  Areas  each  with  their  own  research  objectives  and 
research  threads3 4.  In  the  figures  below  those  task  areas  with  potential  applicability  to  the  Joint  Data  Management 
program  are  highlighted  in  green. 


3  McDermott,  P.,  &  Allender,  L.  (Eds.).  (2009).  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations 
and  Technology:  48hrBooks  by  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD 

21005,  Human  Research  and  Engineering  Directorate,  http://www.arladacta.info. 
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1.2.1.  Task  Area  1:  Cognitive  Modeling  and  Metrics. 

Research  objectives: 

•  Design,  test,  and  transition  innovations  to  facilitate  better  Soldier  understanding  of  the  tactical  situation, 
more  thorough  evaluation  of  courses  of  action,  and,  ultimately,  better  and  more  timely  decisions 

Accomplishments: 

•  Developed  an  efficient  computational  model  of  recognition-primed  decision  making 

•  Created  models  to  describe  interactions  between  fusion  processes  and  decision  making 

•  Conducted  an  experiment  to  determine  how  decision  making  intelligent  agents  can  best  assist  S2  and  S3  in 
the  3 -Block  Challenge  scenario 


C  Aim'L'ptti.il  Models  of  Cognition 

Macrocognitive  Processes 

"■c . ■■■■■■ mmm . f 

Monitoring  and  Remembering 

\  \  \  \  \  v  | 

Learning  with  Automatic  Training  Systems 


YwComputational  Models  of  Cognition 
Computational  Models  ot  SA 


Merged, 
TA2  task 


Analysis  of  Cognitive 

and  Perceptual  Processes  < 

\  \  \  \  \  I 

Learning  and  Comprehension  in  Battle  Situations 


Cognitive  Processes  as  Keys 
to  Battlefield  Performance 


Mental  Projection 

Uncertainty 

’s 


/ZT ^pfenning 


Extending  the  Computational  Model  of  RPD 


Task  Structure  Analysis 

-VvW\ 

IMPRINT  and  ACT-R 


Merged, 
TAJ  task 


Aspects  of  Design 


Animocks 


yj^^hodologies:  Cognitive  Requirements  | 

CDM  Adaptations  Cognitive  Task  Analysis  Text 

\  \  \  \  v.  i  m 

Cognitive  Task  Analysis  of  Army  Operations 


Metrics  for  Evaluating 
Performance  Gains  from 
Fieldable  AD  As 


Models  of  Sequential  Behavior 

1  i i i ^ 


J 


Decision  Centered  Design 

User-Centered  Design  Concepts  tor  Army  Operations 

\ \ \\\  i 

Decision  Centered  Design 

\  \  W\  ‘  I 

Guidance  for  the  Alliance  Effort 

\ 1 \  \  N  I 


Synthetic  ’ 
Task 

Environment 


FY  09 


New  in  09:  Emergent 
Complexity,  Model 
Comparison 

New  in  07: An  Actionable 
Model  of  SA  using  Fuzzy 
Cognitive  Mapping 


Through  09:  Modeling  and 
Measuring  SA 


GrBIL:  Continued  thru  09 


Through  09 


Through  09  (in  TA4) 


Through  09: 
Consolidation  effort 


!  Through  09: 


Figure  2-  Cognitive  Modeling  and  Metrics 


Summary  of  Research  Threads: 

-  Conceptual  Models  of  Cognition. 

-  Cognitive  Processes  as  Keys  to  Battlefield  Performance. 

-  Computational  Models  of  Cognition. 


4  ADA  CTA  Briefing  by  Patty  McDermott,  Alion  Science  and  Technology,  Inc.  (2010) 
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-  Aspects  of  Design. 

-  Methodologies:  Cognitive  Requirements. 

-  Toward  Cognitive  AD  As. 

-  Decision  Centered  Design.: 

1.2.2.  Task  Area  2:  Team  Communication  and  Collaboration. 

Research  objectives: 

•  Create  tools  and  intelligent  systems  that  support  collaborative  decision  making  across  distributed  teams. 

•  Exploit  information  operations  to  provide  data  in  meaningful  ways  to  support  shared  SA,  coordination,  and 
adaptation. 

Accomplishments: 

•  Improved  distributed  planning  activities  via  Collaborative  Slide  Annotation  Tool  during  CERDEC  JF 
Experiment. 

•  Developed  C2  network  analysis  toolkit  to  help  commanders  visualize  C2  structures,  analyze 
effectiveness  and  redesign  the  structure,  if  needed. 

•  Developed  mission  planning  and  replanning  tools. 

•  Specified  measures  to  predict  team  performance  and  subsequent  impact  on  system  performance. 


FYD2 


FVQ3  FYO-4  FYOS 


FYQB 


FY  09 


Collaboration  and  Effectiveness  in 
Human  System  Teamwork 


\\ 

Research  on  Individuals,  Groups,  and  Culture 

T 


T\ 
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-V 
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\  \  \  \ 
Team  Adaptability 
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\  \  \  \ 


QL>ols  to  Help  Research  Teams 
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\  \  \  X  \ 

Team  Task  Network 
\  \  \  X  X 

Team  Training  Laboratory 
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LSA  and 
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Networks 


Builds  on 
iRA  (TA 1) 
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OFR  (TAl) 


Research  on  Culture,  Collaboration 
and  Effectiveness  in  Teamwork 

Facilitating  Multicultural  Collaboration 


Adaptability 
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tural  Teams  A 
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T and  Vulnerabilities  in  C3I  Structures _ 


Team  metacognition  to  improve 
human  to  human  and  human  to 


•velopment  of  Tools  to  Support 
■ration  and  Decision  Making  in 
d  and  Distributed  Teams 


Builds  on 

Synergy 

(TAl) 


Tools  to  Support  Team  SA  in  Collaborative  Tasks 

\  \  \ \\  I 


Interactive  Planning,  Adaptation, 
and  Common  Ground 

\  \  \  \  x  i 

Continuous  Battlefield  Planning  Technology 
\  \  \  X  X  I 

Decision  Support  for  Information 
Gathering  and  Interpretation 


Development  of  Tools  and  Guidelines 
to  Support  Collaboration  and 
Decision  Making 

Shared  SA  in  Ad  Hoc  Teams 


Continuous  Distributed  Planning 


|  Decisioi^Architecture^^searcl^nvironmen^DAI^)^ 


Through  09 


New  in  09:  Ethnographic 
Methods  to  ID  Socio¬ 
tech  nical  Patterns 

Through  09 


Through  09:  Human- 

Automation 

Collaboration 


Through  09 

Through  09:  Plan 
adaption  and  execution 


Figure  3-  Team  Communication  and  Collaboration 


Summary  of  Research  Threads: 

-  Collaboration  and  Effectiveness  in  Human  System  Teamwork. 

-  Research  on  Culture,  Collaboration  and  Effectiveness  in  Teamwork. 
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-  Tools  to  Help  Research  Teams. 

-  Development  of  Tools  to  Support  Collaboration  and  Decision  Making  in  Collocated  and  Distributed  Teams. 

-  Development  of  Tools  and  Guidelines  to  Support  Collaboration  and  Decision  Making. 

1.2.3.  Task  Area  3:  Context-Sensitive  Information  Presentation  (Formerly  User- Adaptable  Interfaces). 

Research  objectives: 

•  Explore  ideas,  frameworks  and  technologies  to  assist  Army  decision  makers  in  getting  the  information 
needed  to  make  sound  decisions. 

•  Allow  Soldier  to  perceive  and  understand  battlefield  information  without  experiencing  cognitive  overload. 
Accomplishments: 

•  Created  haptic  displays  and  guidelines  for  their  use. 

•  Prototyped  adaptive  delegation  interface  to  accomplish  human  supervision  of  multiple  autonomous  agents. 

•  Report  documenting  how  flexible  displays  can  optimally  provide  dismounted  Soldier  information 
requirements. 


\  \  \  v 


Adaptive  Human-Machine 
Cognitive  Systems 


,  ,,  .  ,  Collaborative  8  Middleware  and  [Network  Routine 

|  E«"‘  ,mo  l«lorni4C.on  "  Human-Agent  Teams  A  - - J  L.m-r-.  to  Improve i  w.ll  Adspub.l.u 

lenitive  Field  Research  to  Identic  Difficult  Tasks 


Coj 


Supporting  Communication  through  Integrated  Multimodal 
Information  Presentation 
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Logical  Analysis  ol  Trust 

\  \  \  \ 

_ \ _ \ _ \ _ \ 


\  \  \  N 
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Agent-Human  Interaction  in  Dynamic  Environments 

_ 1 _ \ _ 1 


FY  09 

Agile  Computing 
Infrastructure 

Through  09:  Cross-layer 
Substrate  (in  TA  4) 


Through  09:  Managing 
Uncertainty 

Through  09:  Projecting 
Technology  Adaptation 

Through  09 

Through  09:  Leveraging 
Agile  Computing  (in  TA4) 


Through  09:  Combined 
with  Robotics  projects 

Through  09 
Through  09  (in  TA4) 


Information  Integration 

Info  to  Coordinate 
Perspectives 

Host  Human-Robot 
Interaction  workshop 


Figure  4-  Context  Sensitive  Information  Presentation 


Summary  of  Research  Threads: 

-  Adaptive  Human-Machine  Cognitive  Systems. 

-  Information  Gathering. 

-  Information  Integration. 
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-  Information  Representation  and  Exchange. 

-  Agile  Computing  Infrastructure. 

-  Information  Integration. 

-  Information  to  Coordinate  Perspectives. 

1.2.4.  Task  Area  4:  Fusion  and  Intelligent  Architectures  (Formerly  Auto-Adaptive  Interfaces). 

Research  objectives: 

•  Explore  ideas,  frameworks  and  technologies  to  assist  Army  decision  makers  in  getting  the  information 
needed  to  make  sound  decisions. 

•  Integrate  sophisticated  human  and  machine  capabilities  while  overcoming  their  limits  in: 

o  Information  gathering, 
o  Information  integration, 
o  Information  representation  and  exchange. 

Accomplishments: 

•  Demonstrated  agile  software  agents  on  FCS  platforms. 

•  Demonstrated  computer  reasoning  algorithms  that  address  entity  re-identification  relevant  to  intelligence 
analysis  systems. 

•  Conducted  experiments  to  improve  appropriate  perception  of  risk  in  decisions  that  involve  uncertain 
information  (including  asset  health,  status  and  location). 
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Summary  of  Research  Threads: 

-  Visual  Representations  for  Decision  Assistance. 

-  Interfaces  for  Visual  Decision  Assistance. 

-  Ontology  and  Inferencing  for  Natural  language  Interfaces. 

-  Complementing  the  Visual:  Tactile  and  Language  Interfaces. 

-  Multimodal  Representations  and  Interactions. 

-  Intelligent  Architectures  for  Fusion  and  Planning. 

1.3.  Document  Types. 

There  are  548  study  reports  from  the  ADA  CTA  that  are  documented  in  this  bibliography.  These  reports  and 
documents  consist  of  a  variety  of  formats  and  types.  They  are: 

1.3.1.  Published  reports  from  DTIC. 

These  documents  have  been  published  by  the  originating  organization  in  the  appropriate  technical  report  format  for 
that  organization  and  submitted  to  the  DTIC  for  inclusion  in  the  technical  reports  national  archive.  Each  of  these 
reports  contains  a  reference  to  the  DTIC  control  number  so  it  can  be  accessed  by  qualified  government  and  other 
personnel  in  the  archive. 

1.3.2.  Published  reports  government  agencies. 

These  documents  have  been  published  in  a  technical  report  format  by  the  U.S.  Army  Research  Laboratory’s  Human 
Research  and  Engineering  Directorate. 

1.3.3.  Journal  Articles. 

These  are  documents  published  in  peer  reviewed  technical  journals  and  other  publications. 

1.3.4.  Briefings  from  conference  proceedings. 

These  reports  are  those  presented  at  professional  conferences  and  published  in  the  proceedings  from  that  conference. 

1.3.5.  Book  Chapters. 

These  study  reports  are  those  published  as  a  chapter  in  an  edited  book. 

1.3.6.  Books. 

These  study  reports  are  those  that  resulted  in  the  publishing  of  a  complete  book  on  the  subject. 

1.4.  Report  availability. 

Most  of  the  documents  listed  in  this  bibliography  can  be  obtained  from  multiple  sources.  All  documents  with  a 
DTIC  control  number  can  be  obtained  directly  from  NTIS  either  as  a  PDF  document  that  can  be  downloaded 
directly,  or  in  hardcopy  form  for  a  processing  fee.  Reports  from  the  psychology  based  literature  can  be  obtained 
from  the  publication  source  contained  in  the  report  citation. 

1.5.  How  To  Use  This  Bibliography. 

This  report  is  designed  to  facilitate  both  manual  use  and  automated  searches  of  computer  databases. 

1.5.1.  Using  the  electronic  version  of  the  bibliography. 

This  report  is  structured  so  that  it  can  be  effectively  used  in  its  hardcopy  format  but  it  is  designed  for  optimal  use  in 
electronic  format.  Attached  to  the  back  cover  of  the  hardcopy  version  of  the  report  is  a  CD  containing  an  electronic 
copy  of  this  report  in  a  single  PDF  file.  This  PDF  file  supports  electronic  searches  to  quickly  locate  a  desired  study 
report  using  an  electronic  search  of  the  document. 

1.5.2.  Using  the  hardcopy  version  of  the  bibliography. 

If  the  PDF  version  of  the  bibliography  is  not  available  then  the  reference  listings  in  the  hardcopy  version  of  the 
report  can  be  used  to  support  a  manual  search  for  the  desired  study  report(s). 
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Table:  Format  of  the  Annotated  Report  Listings  in  this  Bibliography _ 

Citation  in  Human  Factors  Format:  Authors  (Year)  Title,  Publisher,  Place  of  publication 
Key  Words: 

Key  words  contained  in  the  report.  If  there  were  no  key  words  in  the  report  then  a  listing  of  categorical  keys  are 
listed. 

Summary  abstract: 

The  abstract  or  introduction  from  the  report. 


1.6.  Distribution  Restrictions  for  this  Bibliography. 

This  document  is  unclassified  and  has  no  distribution  restrictions  and  is  authorized  for  public  release  as  it  contains 
only  cited  references  and  abstracts  from  the  study  reports  listed.  However,  many  of  the  study  reports  cited  may  have 
associated  copyright  limitations  according  to  the  source  of  their  publication. 


2.  Listings  of  Reports  in  This  Bibliography  Grouped  By  Year  of  Publication. 

This  section  provides  a  reference  listing  of  all  the  reports  produced  by  the  ADA  CTA  during  the  course  of  the  life  of 
the  project  from  2001  to  2009.  There  were  some  reports  that  were  not  published  until  2010  after  the  CTA  had  ended 
and  these  are  listed  in  that  year  group.  The  reports  in  each  year  grouping  are  in  order  by  the  primary  author’s  last 
name. 
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2.1.  ADA  CTA  Study  Reports  Published  in  2001. 

Endsley,  M.  R.  (2001).  Designing  for  Situation  Awareness  in  Complex  Systems.  In  Proceedings  of  Second 

International  Workshop  on  Symbiosis  of  Humans,  Artifacts  and  Environment,  pp.  176-190.  Kyoto,  Japan: 
Program  of  the  Japan  Society  for  the  Promotion  of  Science. 

Endsley,  M.  R.,  &  Jones,  D.  G.  (2001).  Disruptions,  Interruptions  and  Information  Attack:  Impact  on  Situation 
Awareness  and  Decision  Making.  In  Proceedings  of  45th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  Minneapolis,  MN  (pp.  63-67). 

Endsley,  M.  R.,  &  Jones,  D.  G.  (2001).  Disruptions,  Interruptions,  and  Information  Attack:  Impact  on  Situation 
Awareness  and  Decision  Making.  In  Proceedings  of  45th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  Minneapolis,  MN. 

Iyer,  N.  S.,  &  Josephson,  J.  R.  (2001).  Multicriterially  Best  Explanations.  In  K.P.  Jantke  and  A.  Shinohara  (Eds.), 
Discovery  Science,  Lecture  Notes  in  Artificial  Intelligence  2226,  Pp.  128-140.  Also  in  Proceedings  of 
Discovery  Science  2001:  The  4th  International  Conference  on  Discovery  Science.  Washington  Dc,  2002. 
New  York,  NY:  Springer  Verlag.  ISBN  3-540-42956-5. 

Kiekel,  P.  A.,  Cooke,  N.  J.,  Foltz,  P.  W.,  &  Shope,  S.  M.  (2001).  Automating  Measurement  of  Team  Cognition 

through  Analysis  of  Communication  Data.  In  M.J.  Smith,  G.  Salvendy,  D.  Harris,  &  R.J.  Koubek  (Eds.), 
Usability  Evaluation  and  Interface  Design.  Pp.  1382  -  1386.  Mahwah,  NJ:  Lawrence  Erlbaum  Associates, 
Inc. 

Laughery,  K.  R.,  Jr.,  Archer,  S.,  &  Corker,  K.  (2001).  Chapter  93.  Modeling  Human  Performance  in  Complex 
Systems.  In  G.  Salvendy  (Ed.),  Handbook  of  Industrial  Engineering  (3  ed.,  pp.  2409-2443).  New  York: 
John  Wiley  &  Sons,  Inc. 

Martinez,  S.  G.,  Talcott,  C.,  Bennett,  K.  B.,  Stansifer,  C.,  &  Shattuck,  L.  (2001).  Cognitive  Systems  Engineering 

Analyses  for  Army  Tactical  Operations.  In  Proceedings  of  45th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  Minneapolis,  MN. 

Riley,  J.  M.,  &  Kaber,  D.  B.  (2001).  Utility  of  Situation  Awareness  and  Attention  for  Describing  Telepresence 
Experiences  in  a  Virtual  Teloperation  Task.  In  Proceedings  of  International  Conference  on  Computer- 
Aided  Ergonomics  and  Safety. 

Stansifer,  C.,  Bennett,  K.  B.,  Talcott,  C.,  Martinez,  S.,  &  Shattuck,  L.  (2001).  Digital  Interfaces  for  Army  Tactical 
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Minneapolis,  MN. 
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Human  Factors  and  Ergonomics  Society.  Minneapolis,  MN. 
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Tools.  In  Proceedings  of  2001  NATO  RTA  Conference  on  Future  Modeling  and  Simulation  Challenges. 
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2.2.  ADA  CTA  Study  Reports  Published  in  2002. 
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SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Warwick,  W.  (2009).  Comparing  the  Comparisons.  Proceedings  of  the  18th  Conference  on  Behavior  Representation 
in  Modeling  and  Simulation  (BRIMS),  Sundance,  UT,  31  March  -  2  April  2009. 
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Warwick.  W.,  Allender,  L.,  &  Yen,  J.  (2009).  Editors’  Introduction  to  the  Special  Issue  on  Developing  and 

Understanding  Computational  Models  of  Macrocognition.,  Special  Section  of  the  Journal  of  Cognitive 
Engineering  and  Decision  Making:  Human  Factors  and  Ergonomics  Society  (HFES). 

Warwick,  W.,  &  Santamaria,  A.  (2009).  Chapter  15.  A  Computational  Model  of  Naturalistic  Decision  Making  and 
the  Science  of  Simulation.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for 
the  Warfighter:  Foundations  and  Technology.  Section  III.  Acting  on  Battlefield  Information:  48hrBooks 
by  Alion  Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from 
the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final 
Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July 
22-23,  2009.  1100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr- 
webtest.  alionscience.com/AdaCta/ Agenda,  aspx. 

Woods,  D.  D.  (2009).  Creating  Foresight:  Lessons  for  Enhancing  Resilience  from  Columbia.  In  B.  Starbuck  and  M. 
Farjoun  (Eds.),  Organization  at  the  Limit:  Nasa  and  the  Columbia  Disaster:  Blackwell,  (in  press). 

Yang,  G.-H.,  Kwon,  D.-S.,  &  Jones,  L.  A.  (2009).  Spatial  Acuity  and  Summation  on  the  Hand:  The  Role  of  Thermal 
Cues  in  Material  Discrimination.  Attention,  Perception  &  Psychophysics.  The  Psychonomic  Society,  Inc., 
71(1),  156-163. 

Yen,  J.,  Strater,  L.,  McNeese,  M.,  Fan,  X.,  Cuevas,  H.  M.,  Oh,  S.,  Kim,  A.,  Minotra,  D.,  &  Hanratty,  T.  (2009). 
Chapter  12.  Cognitively-Inspired  Agents  as  Teammates  and  Decision  Aids.  In  P.  McDermott  and  L. 
Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology. 
Section  II.  Presenting  Battlefield  Information  to  Warfighters:  48hrBooks  by  Alion  Science  &  Technology 
Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005, 
Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures 
(ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting 
and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue 
SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 


64 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 


2.10.  ADA  CTA  Study  Reports  Published  in  2010. 

Allender,  L.  (2010).  Chapter  17.  A  Cognitive  Systems  Engineering  Approach  for  Human-Robot  Interaction: 

Lessons  from  an  Examination  of  Temporal  Latency.  In  M.  Barnes  and  F.  Jentsch  (Eds.),  Human-Robot 
Interactions  in  Future  Military  Operations:  Ashgate  Publishing,  Inc. 

Cooke,  N.  J.,  &  Chadwick,  R.  A.  (2010).  Chapter  19.  Lessons  Learned  from  Human-Robotic  Interaction  on  the 
Ground  and  in  the  Air.  In  M.  Barnes  and  F.  Jentsch  (Eds.),  Human-Robot  Interactions  in  Future  Military 
Operations:  Advanced  Decision  Architectures,  Collaborative  Technology  Alliance. 

Cox,  D.  A.,  &  Veinott,  E.  S.  (2010).  Ethno-Social  Framework  for  Evaluation  User’s  Guide. 

Gillan,  D.  J.,  Riley,  J.,  &  McDermott,  P.  (2010).  Chapter  4.  The  Cognitive  Psychology  of  Human-Robot  Interaction. 
In  M.  Barnes  and  F.  Jentsch  (Eds.),  Human-Robot  Interactions  in  Future  Military  Operations:  Ashgate 
Publishing,  Inc. 

Pazuchanics,  S.  L.,  Chadwick,  R.  A.,  Sapp,  M.  V.,  &  Gillan,  D.  J.  (2010).  Chapter  6.  Robots  in  Space  and  Time: 

The  Role  of  Object,  Motion  and  Spatial  Perception  in  the  Control  and  Monitoring  of  UGVs.  In  M.  Barnes 
and  F.  Jentsch  (Eds.),  Human-Robot  Interactions  in  Future  Military  Operations:  New  Mexico  State 
University. 

Riley,  J.  M.,  Strater,  L.  D.,  Chappell,  S.  L.,  Connors,  E.  S.,  &  Endsley,  M.  R.  (2010).  Chapter  10.  Situation 

Awareness  in  Human-Robot  Interaction:  Challenges  and  User  Interface  Requirements.  In  M.  Barnes  and  F. 
Jentsch  (Eds.),  Human-Robot  Interactions  in  Future  Military  Operations.  London:  Ashgate  Publishing,  Inc. 

Thompson,  L.  F.,  &  Gillan,  D.  J.  (2010).  Chapter  5.  Social  Factors  in  Human-Robot  Interaction.  In  M.  Barnes  and 
F.  Jentsch  (Eds.),  Human-Robot  Interactions  in  Future  Military  Operations.  London:  Ashgate  Publishing, 
Inc. 

Veinott,  E.  S.,  Klein,  G.  A.,  &  Wiggins,  S.  L.  (2010).  Evaluating  the  Effectiveness  of  the  Premortem  Technique  on 
Plan  Confidence. 

Woods,  D.  D.,  &  Sarter,  N.  B.  (2010).  Capturing  the  Dynamics  of  Attention  Control  from  Individual  to  Distributed 
Systems:  The  Shape  of  Models  to  Come  Theoretical  Issues  in  Ergonomics  Science  (Special  Issue  on 
Situation  Awareness).  11(1-2),  7-28. 


65 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 


This  page  left  blank 


66 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 

3.  The  Annotated  Bibliography. 

This  section  is  the  main  body  of  this  report.  All  of  the  reports  listed  are  in  order  by  primary  author. 

Agrawal,  A.,  Meth,  R.,  &  Chellappa,  R.  (2005).  Hierarchical  DEM  Refinement  Using  Surface  Parallax.  In 
Proceedings  of  24th  Army  Science  Conference.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

We  present  a  multi-resolution  approach  to  update  and  refine  coarse  3D  models  of  urban  environments  from  a 
sequence  of  intensity  images  using  surface  parallax.  A  coarse  and  potentially  incomplete  depth  map  of  the  scene 
obtained  from  a  Digital  Elevation  Map  (DEM)  is  used  as  a  reference  surface  which  is  refined  and  updated  using  this 
approach.  We  first  estimate  the  camera  motion  using  the  reference  depth  map.  Using  the  estimated  camera  motion, 
at  each  level  in  the  multi-resolution  framework,  motion  of  3D  points  on  the  reference  surface  is  compensated,  and 
the  residual  flow  field,  which  is  an  epipolar  field,  is  estimated  and  used  to  refine  the  depth  map  at  that  level.  At  a 
coarse  resolution,  the  difference  between  the  reference  depth  and  the  true  depth  will  be  small,  leading  to  a  small 
parallax  field.  The  refined  depth  map  from  the  coarser  level  is  then  propagated  to  the  finer  level  and  is  used  as  a 
reference  depth  map  at  that  level.  Thus,  significant  deviations  of  an  available  model  from  a  true  model  can  be 
handled  using  this  approach. 


Agrawal,  A.,  &  Chellappa,  R.  (2005).  Moving  Object  Segmentation  and  Dynamic  Scene  Reconstruction  Using  Two 
Frames.  In  In  Proceedings  of  IEEE  International  Conference  on  Acoustics,  Speech,  and  Signal  Processing.  (Winner 
of  student  paper  award.). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

We  present  an  iterative  algorithm  for  segmenting  independently  moving  objects  and  refining  and  updating  a  coarse 
depth  map  of  the  scene  under  unconstrained  camera  motion  (translation  and  rotation)  with  the  assumption  that  the 
independently  moving  objects  undergoes  pure  translation.  Given  a  coarse  depth  map  acquired  by  a  range-finder  or 
extracted  from  a  Digital  Elevation  Map  (DEM),  the  ego-motion  is  estimated  by  combining  a  global  ego-motion 
constraint  and  a  local  brightness  constancy  constraint  using  least  median  of  squares  (LMedS)  which  treats 
independently  moving  objects  as  outliers.  Using  the  estimated  camera  motion  and  the  available  depth  estimate, 
motion  of  the  3D  points  is  compensated.  We  utilize  the  fact  that  the  resulting  surface  parallax  field  is  an  epipolar 
field  and  use  a  corresponding  parametric  model  to  estimate  the  parallax  vectors  for  all  pixels.  We  use  the  previous 
motion  estimate  to  get  the  epipolar  direction  and  hence  pixels  where  the  parallax  direction  is  not  aligned  towards  the 
epipolar  direction  are  segmented  out  as  moving  points.  The  depth  map  for  static  pixels  is  refined  using  the  estimated 
parallax  vectors.  All  segmented  regions  are  removed  for  robustly  estimating  the  ego-motion  in  subsequent  iterations. 
A  parametric  flow  model  is  fitted  to  the  segmented  regions  and  their  3D  motion  is  estimated  using  subspace 
analysis.  We  present  experimental  results  using  both  synthetic  and  real  data  to  validate  the  effectiveness  of  the 
proposed  algorithm. 


Agrawal,  A.,  &  Chellappa,  R.  (2005).  Robust  Ego-Motion  Estimation  and  3D  Model  Refinement  in  Scenes  with 
Varying  Illumination.  In  In  Proceedings  of  IEEE  Computer  Society  Workshop  on  Motion  and  Video  Computing. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

We  present  an  iterative  algorithm  for  robustly  estimating  the  ego-motion  and  refining  and  updating  a  coarse  depth 
map  using  surface  parallax  and  a  generalized  dynamic  image  (GDI)  model.  Given  a  coarse  depth  map  acquired  by  a 
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range-finder  or  extracted  from  a  Digital  Elevation  Map  (DEM),  we  first  estimate  the  ego-motion  by  combining  a 
global  ego-motion  constraint  and  a  local  GDI  model.  Using  the  estimated  camera  motion  and  the  available  depth 
estimate,  motion  of  the  3D  points  is  compensated.  We  utilize  the  fact  that  the  resulting  surface  parallax  field  is  an 
epipolar  field  and  constrain  its  direction  using  the  previous  motion  estimates.  We  then  estimate  the  magnitude  of  the 
parallax  field  and  the  GDI  model  parameters  locally  and  use  them  to  refine  the  depth  map  estimates.  We  use  a  tensor 
based  approach  to  formulate  the  depth  refinement  procedure  as  an  eigen-value  problem  and  obtain  confidence 
measures  for  determining  the  accuracy  of  the  estimated  depth  values.  These  confidence  measures  are  used  to  remove 
regions  with  potentially  incorrect  depth  estimates  for  robustly  estimating  ego-motion  in  the  next  iteration. 
Experimental  results  using  both  synthetic  and  real  data  are  presented.  Comparisons  with  results  obtained  using  a 
brightness  constancy  (BC)  model  show  that  the  proposed  algorithm  works  significantly  better  when  time-varying 
illumination  changes  are  present  in  the  scene. 


Agrawal,  A.,  &  Chellappa,  R.  (2005).  Robust  Ego-Motion  Estimation  and  3D  Model  Refinement  Using  Depth 
Based  Parallax  Model.  In  In  Proceedings  of  IEEE  International  Conference  on  Image  Processing. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

We  present  an  iterative  algorithm  for  robustly  estimating  the  egomotion  and  refining  and  updating  a  coarse,  noisy 
and  partial  depth  map  using  a  depth  based  parallax  model  and  brightness  derivatives  extracted  frnm  an  image  pair. 
Given  a  coarse,  noisy  and  partial  depth  map  acquired  by  a  range-tinder  or  obtained  from  a  Digital  Elevation  Map 
(DEM),  we  first  estimate  the  ego-motion  by  combining  a  global  ego-motion  constraint  and  a  local  brightness 
constancy  constraint.  Using  the  estimated  camera  motion  and  the  available  depth  map  estimate,  motion  of  the  3D 
points  is  compensated.  We  utilize  the  fact  that  the  resulting  surface  parallax  field  is  an  epipolar  field  and  knowing 
its  direction  from  the  previous  motion  estimates,  estimate  its  magnitude  and  use  it  to  refine  the  depth  map  estimate. 
Instead  of  assuming  a  smooth  parallax  field  or  locally  smooth  depth  models,  we  locally  model  the  parallax 
magnitude  using  the  depth  map,  formulate  the  problem  as  a  generalized  eigen-value  analysis  and  ohtain  better 
results. 

In  addition,  confidence  measures  for  depth  estimates  are  provided  which  can  he  used  to  remove  regions  with 
potentially  incorrect  (and  outliers  in) 

depth  estimates  for  robustly  estimating  ego-motion  in  the  next  iteration.  Results  on  both  synthetic  and  real  examples 
are  presented. 


Agrawal,  A.,  &  Chellappa,  R.  (2006).  Fusing  Depth  and  Video  Using  Rao-Blackwellized  Particle  Filter.  In  In 
Proceedings  of  First  International  Conference  on  Pattern  Recognition  and  Machine  Intelligence,  Kolkata,  India. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

We  address  the  problem  of  fusing  sparse  and  noisy  depth  data  obtained  from  a  range  finder  with  features  obtained 
from  intensity  images  to  estimate  ego-motion  and  refine  3D  structure  of  a  scene  using  a  Rao-Blackwellized  particle 
Iter.  For  scenes  with  low  depth  variability,  the  algorithm  shows  an  alternate  way  of  performing  Structure  from  Mo¬ 
tion  (SfM)  starting  with  a  °at  depth  map.  Instead  of  using  3D  depths,  we  formulate  the  problem  using  2D  image 
domain  parallax  and  show  that  conditioned  on  non-linear  motion  parameters,  the  parallax  magni-tude  with  respect  to 
the  projection  of  the  vanishing  point  forms  a  linear  subsystem  independent  of  camera  motion  and  their  distributions 
can  be  analytically  integrated.  Thus,  the  structure  is  obtained  by  estimating  parallax  with  respect  to  the  given  depths 
using  a  Kalman  filter  and  only  the  ego-motion  is  estimated  using  a  particle  filter.  Hence,  the  required  number  of 
particles  becomes  independent  of  the  number  of  feature  points  which  is  an  improvement  over  previous  algorithms. 
Experimental  results  on  both  synthetic  and  real  data  show  the  e®ectiveness  of  our  approach. 
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Agrawal,  A.,  &  Chellappa,  R.  (2006).  Robust  Ego-Motion  Estimaton  and  3D  Model  Refinement  Using  Surface 
Parallax.  IEEE  Transactions  on  Image  Processing,  May  2006,  15(5),  1215  -  1225. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Direct  methods,  surface  parallax,  three-dimensional  (3-D)  modeling. 

Summary  abstract: 

We  present  an  iterative  algorithm  for  robustly  estimating  the  ego-motion  and  refining  and  updating  a  coarse  depth 
map  using  parametric  surface  parallax  models  and  brightness  derivatives  extracted  from  an  image  pair.  Given  a 
coarse  depth  map  acquired  by  a  range-finder  or  extracted  from  a  digital  elevation  map  (DEM),  ego-motion  is 
estimated  by  combining  a  global  ego-motion  constraint  and  a  local  brightness  constancy  constraint.  Using  the 
estimated  camera  motion  and  the  available  depth  estimate,  motion  of  the  three-dimensional  (3-D)  points  is 
compensated.  We  utilize  the  fact  that  the  resulting  surface  parallax  field  is  an  epipolar  field,  and  knowing  its 
direction  from  the  previous  motion  estimates,  estimate  its  magnitude  and  use  it  to  refine  the  depth  map  estimate.  The 
parallax  magnitude  is  estimated  using  a  constant  parallax  model  (CPM)  which  assumes  a  smooth  parallax  field  and  a 
depth  based  parallax  model  (DBPM),  which  models  the  parallax  magnitude  using  the  given  depth  map.  We  obtain 
confidence  measures  for  determining  the  accuracy  of  the  estimated  depth  values  which  are  used  to  remove  regions 
with  potentially  incorrect  depth  estimates  for  robustly  estimating  ego-motion  in  subsequent  iterations.  Experimental 
results  using  both  synthetic  and  real  data  (both  indoor  and  outdoor  sequences)  illustrate  the  effectiveness  of  the 
proposed  algorithm. 


Agrawal,  A.,  Raskar,  R.,  &  Chellappa,  R.  (2006).  Edge  Suppression  by  Gradient  Field  Transformation  Using  Cross- 
Projection  Tensors.  In  In  Proceedings  of  the  2006  IEEE  Computer  Society  Conference  on  Computer  Vision  and 
Pattern  Recognition  (CVPR’06). 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Conference  Proceedings 

Summary  abstract: 

We  propose  a  new  technique  for  edge-suppressing  operations  on  images.  We  introduce  cross  projection  tensors  to 
achieve  affine  transformations  of  gradient  fields.  We  use  these  tensors,  for  example,  to  remove  edges  in  one  image 
based  on  the  edge-information  in  a  second  image.  Traditionally,  edge  suppression  is  achieved  by  setting  image 
gradients  to  zero  based  on  thresholds.  A  common  application  is  in  the  Retinex  problem,  where  the  illumination  map 
is  recovered  by  suppressing  the  reflectance  edges,  assuming  it  is  slowly  varying. 

We  present  a  class  of  problems  where  edge-suppression  can  be  a  useful  tool.  These  problems  involve  analyzing 
images  of  the  same  scene  under  variable  illumination.  Instead  of  resetting  gradients,  the  key  idea  in  our  approach  is 
to  derive  local  tensors  using  one  image  and  to  transform  the  gradient  field  of  another  image  using  them. 
Reconstructed  image  from  the  modified  gradient  field  shows  suppressed  edges  or  textures  at  the  corresponding 
locations.  All  operations  are  local  and  our  approach  does  not  require  any  global  analysis. 

We  demonstrate  the  algorithm  in  the  context  of  several  applications  such  as  (a)  recovering  the  foreground  layer 
under  varying  illumination,  (b)  estimating  intrinsic  images  in  non-Lambertian  scenes,  (c)  removing  shadows  from 
color  images  and  obtaining  the  illumination  map,  and  (d)  removing  glass  reflections. 


Agrawal,  A.,  Raskar,  R.,  &  Chellappa,  R.  (2006).  What  Is  the  Range  of  Surface  Reconstructions  from  a  Gradient 
Field?  In  In  Proceedings  of  9th  European  Conference  on  Computer  Vision.  Austria,  Graz. 

Key  Words: 
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ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  psychology  of  remote-operator  environmental  awareness  is  an  important  area  of  inquiry  in  which  recent 
advances  enable  improved  deployment  of  robotic  vehicles.  The  use  of  unmanned  ground  vehicles  (UGVs)  allows 
for  remote  sensing  and  action  in  hazardous  environments  at  a  safe  distance.  Proponents  of  UGV  deployment  for 
both  military  and  civilian  applications  must  overcome  technical  challenges  faced  in  realizing  their  full  potential.  The 
development  of  operationally  effective  UGVs  remains  demanding  due  to  the  complexity  of  autonomous  ground 
navigation  and  the  cognitive  requirements  on  human  operators.  These  difficulties  appear  across  a  variety  of  UGV 
applications,  including  the  use  of  large  autonomous  vehicles  for  reconnaissance  or  convoy  operations  and  the 
deployment  of  small  remotely  operated  vehicles  for  explosives  disposal  or  urban  search-and-rescue  tasks. 


Agrawal,  A.  K.,  &  Chellappa,  R.  (2004).  3D  Model  Refinement  Using  Surface-Parallax.  IEEE  Acoustics,  Speech, 
and  Signal  Processing,  (Vol.  3),  pp.  285-288.  Presented  at  ICASSP  2004.  Montreal,  Canada. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

We  present  an  approach  to  update  and  refine  coarse  3D  models  of  urban  environments  from  a  sequence  of  intensity 
images  using  surface  parallax.  This  generalizes  the  plane-parallax  recovery  methods  to  surface -parallax  using 
arbitrary  surfaces.  A  coarse  and  potentially  incomplete  depth  map  of  the  scene  obtained  from  a  Digital  Elevation 
Map  (DEM)  is  used  as  a  reference  surface  which  is  refined  and  updated  using  this  approach.  The  reference  depth 
map  is  used  to  estimate  the  camera  motion  and  the  motion  of  the  3D  points  on  the  reference  surface  is  compensated. 
The  resulting  parallax,  which  is  an  epipolar  field,  is  estimated  using  an  adaptive  windowing  technique  and  used  to 
obtain  the  refined  depth  map. 


Agrawal,  A.  K.,  Meth,  R.,  &  Chellappa,  R.  (2004).  Surface  Parallax  Based  Hierarchical  Approach  for  Model 
Refinement  (Hierarchical  DEM  Refinement  Using  Surface  Parallax).  In  In  Proceedings  of  2004  ICPR.  Cambridge, 
UK.  DTIC  ADA431639. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Fields  and  Groups: 

080200  -  Cartography  and  Aerial  Photography 
Descriptors: 

(U)  *DIGITAL  MAPS,  SYMPOSIA,  THREE  DIMENSIONAL,  FLOW  FIELDS,  ELEVATION, 

URBAN  AREAS,  DIGITAL  CAMERAS 
Identifiers: 

(U)  COMPONENTS  REPORTS 
Summary  abstract: 

We  present  a  multi-resolution  approach  to  update  and  refine  coarse  3D  models  of  urban  environments  from  a 
sequence  of  intensity  images  using  surface  parallax.  A  coarse  and  potentially  incomplete  depth  map  of  the  scene 
obtained  from  a  Digital  Elevation  Map  (DEM)  is  used  as  a  reference  surface  which  is  refined  and  updated  using  this 
approach.  We  first  estimate  the  camera  motion  using  the  reference  depth  map.  Using  the  estimated  camera  motion, 
at  each  level  in  the  multiresolution  framework,  motion  of  3D  points  on  the  reference  surface  is  compensated,  and  the 
residual  flow  field,  which  is  an  epipolar  field,  is  estimated  and  used  to  refine  the  depth  map  at  that  level.  At  a  coarse 
resolution,  the  difference  between  the  reference  depth  and  the  true  depth  will  be  small,  leading  to  a  small  parallax 
field.  The  refined  depth  map  from  the  coarser  level  is  the  propagated  to  the  finer  level.  Thus,  significant  deviations 
of  an  available  model  from  a  true  model  can  be  handled  using  this  approach. 
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Agrawal,  A.  K.,  Shekhar,  C„  David,  P.,  &  DeHart,  J.  (2003).  Mapping  Ground  Video  to  Aerial  Dem’s.  In  In 
Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the 
Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S. 

Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 

Digital  Elevation  Map,  Mobile  Robot,  Video,  Registration,  Depth  Map,  Simulated  Annealing,  Simplex  algorithm, 

Hausdorff  Distance,  Color  Segmentation 

Summary  abstract: 

We  present  an  approach  for  registering  an  aerial  Digital  Elevation  Model  (DEM)  with  a  color  intensity  image 
obtained  using  a  camera  mounted  on  a  mobile  robot.  An  approximate  measurement  of  the  camera  pose  is  obtained 
using  auxiliary  sensors  on-board  the  robot.  The  DEM  is  transformed  into  a  depth  map  in  the  camera’s  coordinate 
system  using  this  initial  pose.  The  problem  is  now  simplified  to  the  alignment  of  two  images,  one  containing 
intensity  information,  and  the  other,  depth.  Region  boundaries  in  the  intensity  image  are  matched  with 
discontinuities  in  the  depth  map  using  a  robust  directed  Hausdorff  distance.  This  cost  function  is  minimized  with 
respect  to  the  six  parameters  defining  the  camera  pose.  Due  to  the  highly  non-linear  nature  of  cost  function  with 
multiple  local  minima,  a  stochastic  algorithm  based  on  the  downhill  simplex  principle  is  employed  for  minimization. 
Results  on  real  data  are  presented. 


Allender,  L.  (2010).  Chapter  17.  A  Cognitive  Systems  Engineering  Approach  for  Human-Robot  Interaction: 
Lessons  from  an  Examination  of  Temporal  Latency.  In  M.  Barnes  and  F.  Jentsch  (Eds.),  Human-Robot  Interactions 
in  Future  Military  Operations:  Ashgate  Publishing,  Inc. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Advanced  robotics  technology  holds  promise  for  extending,  enhancing,  even  replacing  humans  in  the  performance 
of  dangerous  or  tedious  tasks.  Search  and  rescue  robots  can  go  where  humans  cannot-or  dare  not-and  become 
remote  eyes  and  ears  for  human  first  responders  and  rescue  teams  (Casper  &  Murphy,  2003).  Robots  that  provide 
some  level  of  care  or  supervision  in  medical  contexts  are  being  tested  and  are  intended  to  enhance  the  well  being  of 
the  person  being  cared  for  as  well  as  lightening  the  loads  of  other  care -providers  (e.g.,  Gockley  &  Matari6,  2006). 
Robots  are  even  being  evaluated  in  social  and  educational  settings  with  children  to  understand  the  development,  or 
evolution,  of  naturalistic  human  robot  interaction  (e.g.,  Tanaka  et  aL,  2006).  Of  course,  robots  may  also  be  fun,  such 
as  the  cocktail-serving  robots  offered  in  the  2003  Neiman  Marcus  Christmas  catalog. 


Allender,  L.,  McDermott,  P.  L.,  Luck,  J.,  &  Fisher,  A.  (2006).  Team  Communication  with  and  without  Aids  for 
Transmitting  Remote  Information.  In  In  Proceedings  of  50th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society,  San  Francisco,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  introduction  of  unmanned  vehicles  (UVs)  and  real-time  electronic  information  presentation  to  military  teams  is 
intended  to  keep  Soldiers  safe  and  enable  more  effective  performance.  Two  game-based  experiments  were 
conducted  in  which  teams  conducted  a  time-limited  “Black  Hawk  Down”  rescue  mission  with  and  without  UV- 
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provided  information.  When  UV  information  was  available,  it  was  relayed  between  team  members  in  face-to-face 
communication  or  remotely,  with  or  without  electronic  maps,  and  with  or  without  video  images  from  the  UV.  Three 
types  of  analyses  were  performed.  The  communication  content  of  the  verbal  protocol  was  analyzed  and  the  use  of 
information  display  technologies  and  employment  of  strategies  was  tabulated.  The  verbal  protocol  analyses  revealed 
the  push/pull  of  team  communication  and  common  confusions.  Results  of  the  technology  use  and  strategy 
assessment  form  the  basis  for  recommendations  for  display  design  and  recommended  practice  for  the  use  of  such 
technologies  in  the  field. 


Allsopp,  D.,  Beautement,  P.,  Kirton,  M„  Bradshaw,  J.  M.,  Suri,  N.,  Tate,  A.,  &  Burstein,  M.  (2003).  The  Coalition 
Agents  Experiment:  Network-Enabled  Coalition  Operations.  Special  Issue  on  Network-enabled  Capabilities,  Journal 
of  Defence  Science,  8(3),  130-141. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Multinational  coalitions  are  increasingly  important  in  military  operations.  But  coalitions  today  suffer  from 
heterogeneous  command  systems,  labour-intensive  information  collection  and  coordination,  and  different  and 
incompatible  ways  of  representing  information.  The  purpose  of  Network  Enabled  Capability  (NEC)  is  to  enhance 
military  capability  by  exploiting  information  better.  The  Coalition  Agents  Experiment  (CoAX)  was  an  international 
collaborative  research  effort  to  examine  how  the  emerging  technologies  of  software  agents  and  the  semantic  web 
could  help  to  construct  coherent  command  support  systems  for  coalition  operations.  Technology  demonstrations 
based  on  a  realistic  coalition  scenario  showed  how  agents  and  associated  technologies  facilitated  run-time 
interoperability  across  the  coalition,  responded  well  to  unexpected  battlespace  events,  and  aided  the  selective 
sharing  of  information  between  coalition  partners.  We  describe  the  CoAX  experiments,  the  approaches  and 
technologies  used,  and  highlight  how  they  support  the  NEC  concept.  CoAX  produced  a  prototype  “Coalition  agents 
starter  pack”  that  could  be  developed  further  to  support  coalition  warfare. 


Allsopp,  D.  N.,  Bradshaw,  J.  M.,  Suri,  N.,  Durfee,  E.  H.,  Knoblock,  C.  A.,  Tate,  A.,  &  Thompson,  C.  W.  (2002). 
Coalition  Agents  Experiment:  Multiagent  Cooperation  in  International  Coalitions.  Special  Issue  on  Knowledge 
Systems  for  Coalition  Operations.  IEEE  Intelligent  Systems,  17(3),  26-35. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Military  success  requires  executing  high-tempo,  coherent,  decisive  actions  faster  than  an  opponent  can  react — that 
is,  decision  dominance  through  command  agility.  Command  agility  means  having  the  flexibility  to  grasp  fleeting 
opportunities  and  being  innovative,  creative,  and  unpredictable  in  a  manner  that  (even  if  low-tempo)  increases 
confusion  in  the  opponent’s  mind.  This  process  is  command-led,  with  human  decision  making  primary  and 
technology  playing  a  secondary  role.  Shared  understanding  and  information  superiority  are  key  enablers  in  this 
process  and  are  fundamental  to  network-centric  warfare  (www.dodccrp.org).  In  addressing  interoperability 
requirements,  we  must  also  address  data  security,  control  over  semitmsted  software  from  other  coalition  partners, 
and  the  resulting  system’s  robustness — for  example,  its  ability  to  withstand  denial-of-service  attacks. 

However,  good  decisions  depend  on  good  data,  which  can  become  a  problem  as  mission  complexity  increases. 
Military  coalitions — large-scale,  multifaceted,  multinational,  virtual  organizations — often  must  be  rapidly  created 
and  changed  as  circumstances  shift.  In  addition  to  integrating  single-service  and  joint  capabilities  into  a  coherent 
force,  coalition  operations  must  rapidly  configure  foreign  or  legacy  systems  into  a  cohesive  whole.  Yet  current 
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coalition  operations  often  suffer  from  data  overload,  information  starvation,  laborintensive  information  collection 
and  coordination,  and  stand-alone  stovepipe  command  systems  that  use  incompatible  data  formats.  This  leads  to  a 
horrendous  technical  integration  task  and  offers  commanders  only  scattered  snapshots  of  the  battlespace. 

In  the  inevitable  absence  of  preexisting  coordinated  systems,  we  must  take  a  rapid,  flexible,  on-the-fly  approach  that 
permits  assembling  capabilities  at  runtime.  We  believe  agent-based  computing  (described  in  the  sidebar,  “Software 
Agent  Technology”)  offers  a  promising  new  approach  to  effective  coalition  operations  because  it  embraces  the 
coalition  environment’s  open,  heterogeneous,  diverse,  dispersed  nature.  This  article  describes  how  software  agents 
acting  on  behalf  of  human  users  enable  military  commanders  to  act  decisively  in  cyberspace  and  thus  contribute  to 
“cyberspace  superiority,”  a  critical  component  of  warfare  in  the  information  age.l  We  focus  here  on  the  rapid 
integration  of  agents  and  legacy  systems  to  improve  interoperability  and  support  human  situational  awareness  and 
decision  making,  rather  than,  for  example,  sophisticated  teamwork  and  planning  between  agents.2 


Anders,  S.,  Patterson,  E.  S.,  Woods,  D.  D.,  &  Ebright,  P.  R.  (2007).  Projecting  Trajectories  for  a  New  Technology 
Based  on  Cognitive  Task  Analysis  and  Archetypal  Patterns:  The  Electronic  leu.  In  In  Proceedings  of  K.Mosier  &  U. 
Fischer  (eds.),  Proceedings  of  the  Eighth  International  Conference  on  Naturalistic  Decision  Making. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Ethnographic  observation,  cognitive  task  analysis,  S-curve  for  technology  adoption,  electronic  ICU, 
healthcare,  anomaly  response,  complexity 

Summary  abstract: 

In  many  cases,  cognitive  task  analysis  (CTA)  methods  are  used  to  identify  patterns  in  how  expert  practitioners 
interact  with  computerized  support  to  achieve  domain-specific  objectives.  A  CTA  rarely  makes  explicit  projections 
about  potential  trajectories  of  adaptation  through  use  that  may  occur  over  the  longer  term.  Studies  of  technology 
change  consider  such  trajectories  of  change  and  adaptation  as  new  systems  and  capabilities  are  introduced  and 
mature.  One  model  of  potential  trajectories  of  change  is  depicted  by  the  so-called  “S-curve”  for  diffusion  of 
innovation,  where  new  technologies  are  adopted  early  by  more  educated  users  with  higher  propensity  to  accept  risk 
(“innovators”),  followed  by  accelerated  adoption  for  the  majority  of  users,  and  a  slower  adoption  pace  by  less 
educated,  risk-averse  users  (“laggards”).  We  suggest  that  the  Naturalistic  Decision  Making  (NDM)  research  base 
may  be  sufficient  to  offer  useful  alternatives  to  the  S-curve  model.  We  illustrate  the  proposed  approach  with  a 
telemedicine  example:  the  electronic  Intensive  Care  Unit  (e-ICU).  Based  upon  “bootstrapping”  interviews  and  40 
hours  of  direct  observation  in  one  e-ICU,  we  describe  three  functions  of  the  technology  as  it  is  currently  used:  1) 
anomaly  response,  2)  access  to  specialized  expertise,  and  3)  sensemaking.  We  discuss  how  explicit  projections  may 
support  steering  technology  design  and  implementation  in  more  productive  directions,  and  how  we  might  manage 
negative  “side  effects”  associated  with  particular  trajectories  by  addressing  these  as  post-conditions  to  be  met  as  new 
devices  or  systems  move  toward  realization.  We  conclude  that  the  ability  to  project  possible  trajectories  in  order  to 
aid  in  selecting  among  alternatives  and  managing  post-conditions  of  change  can  be  done  based  on  the  pattern  base 
built  up  through  NDM  work.  Useful  CTAs  should  have  the  ability  to  make  explicit  trajectory  projections  and 
identify  post-conditions. 


Anders,  S.,  Zelik,  D.,  Jacoby,  T.,  Patterson,  E.  S.,  &  Woods,  D.  D.  (2007).  Exploring  Challenges  of  Information 
Dynamics  Using  an  Animock.  In  In  Proceedings  of  51st  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Baltimore,  MD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  this  paper,  we  describe  the  use  of  an  animated  prototyping  technique  to  elicit  feedback  on  the  usefulness  of  a  set 
of  modular  design  concepts  for  dealing  with  information  dynamics  in  inferential  analysis  under  data  overload 
conditions.  The  design  concepts  were  comprised  of  innovative  solutions  that  utilized  technology  to  assist  in 
inferential  analysis.  The  findings  generally  support  the  “promisingness”  of  the  design  directions  for  addressing 
information  dynamics  challenges  in  inferential  analysis  under  data  overload.  Elicited  feedback  provides  insight  on 
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how  the  concepts  might  prove  useful  to  intelligence  analysts  in  the  field.  Analysts  recommended  significant 
modifications  that  would  be  difficult  to  change  post-implementation  of  software,  suggesting  that  the  animock 
technique  was  useful  for  exploring  how  design  concepts  could  address  challenging  issues  where  no  current  software 
support  exists. 

In  this  paper,  we  describe  the  use  of  an  animated  prototyping  technique,  so-called  animocks  (Roesler  et  al.,  2005),  to 
explore  the  usefulness  of  design  concepts  to  address  information  dynamics  challenges.  Specifically,  an  animock  was 
used  in  2.5  hour  focus  group  elicitation  sessions  at  two  intelligence  agencies.  The  animock  illlustrated  a  set  of 
modular  design  concepts,  or  “design  seeds”  (Woods,  1998;  cf.,  Patterson  et  ah,  2001),  that  covered  the  three  high- 
level  cognitive  functions  in  inferential  analysis:  down  collect,  conflict  and  corroboration,  and  hypothesis  exploration 
(Elm  et  ah,  2005).  The  animock  is  a  dynamic  storyboarding  technique  that  represents  a  potential  envisioned  world. 

In  the  current  paper  researchers  present  the  concepts,  findings  from  the  focus  groups,  and  discuss  the  use  of 
animocks  for  eliciting  information  about  envisioned  support  tools  for  particularly  challenging  problems  where  no 
current  software  support  exists. 


Archer,  R.,  Lebiere,  C.,  Warwick,  W.,  Schunk,  D.  W.,  &  Biefeld,  E.  (2003).  Integration  of  Task  Network  and 
Cognitive  Models  to  Support  System  Design.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference 
2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of 
Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

This  paper  describes  the  integration  of  the  IMPRINT  task  network  modeling  tool  with  the  ACT-R  cognitive 
architecture.  The  resulting  system  was  used  to  model  the  interaction  between  an  operator  and  an  interface 
controlling  the  Demo  III  robotic  vehicle.  IMPRINT  was  used  to  represent  the  state  of  the  vehicle  and  the  control 
interface  while  ACT-R  modeled  the  cognitive  state  of  the  operator.  The  combined  model  was  used  to  evaluate  the 
design  of  the  interface  and  especially  the  occurrence  of  errors.  The  model  produced  a  wide  range  of  cognitive  errors, 
and  made  suggestions  for  design  changes  that  could  help  reduce  their  occurrence. 


Archer,  S.,  Archer,  R.,  &  Matessa,  M.  (2007).  Our  GRBIL  Has  a  Split  Personality.  In  In  Proceedings  of  2007  Digital 
Human  Modeling  for  Design  and  Engineering  Conference,  SAE  International,  Seattle,  WA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

DEFINITIONS,  ACRONYMS,  ABBREVIATIONS 

OCU :  Operator  Control  Unit 

ACT  -R:  Adaptive  Control  of  Thought  -  Rational 

VTI:  Vetronics  Technology  Integration 

ARL:  Army  Research  laboratory 

WYSIWYG:  What  you  see  is  what  you  get 

GRB11:  Graph-Based  Interface  language 

Summary  abstract: 

Developers  of  future  systems  are  relying  on  graphical  user  interfaces  (GUI)  for  user  interaction.  As  the  interfaces  get 
more  powerful  and  complex,  it  becomes  increasingly  important  to  be  able  to  predict  how  well  they  will  work  prior 
to  absorbing  the  cost  of  implementation.  This  paper  describes  the  development,  application,  and  future  plans  for  the 
Graph-Based  Interface  language  (GRB11)  tool.  GRB11  can  be  used  to  predict  human  performance  on  new  GUI 
designs,  resulting  in  quantitative  comparisons  between  interface  alternatives.  This  enables  human  factors  engineers 
to  participate  in  system  design  processes  and  achieve  influence  very  early.  Using  GRB11,  an  interface  designer  can 
sketch  the  proposed  interfaces,  describe  the  tasks  that  the  user  wiU  perform,  and  even  inject  stressor  effects  to 
represent  fatigued  or  under-trained  users.  GRB11  will  generate  estimates  for  the  times  to  perform  the  tasks  and  will 
predict  whether  errors  (such  as  those  caused  by  memory  lapses)  are  likely.  The  designer  can  then  use  GRB11  to 
adjust  the  GUI  to  explore  alternatives  that  might  improve  performance. 
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Arguedas,  M.,  Perez,  C.,  Carvalho,  M.,  Granados,  A.,  Hobakc,  K.,  Kraus,  W.  (2009).  Investigating  the  Use  of 
Topology  Adaptation  for  Robust  Multi-Path  Transport:  A  Preliminary  Study.  In  In  Proceedings  of  8th  IEEE 
International  Symposium  on  Network  Computing  and  Applications,  NCA  09,  Boston,  July  9-11,  2009. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  this  paper  we  introduce  a  topology  control  algorithm  for  increasing  transport  robustness  and  efficiency  by 
creating  two  simultaneous  communication  paths  between  the  source  and  destination  nodes  of  a  data  flow.  We  make 
use  of  a  cross-layer  substrate  that  allows  us  to  detect  the  flows  of  data  in  the  network  while  also  providing  finer 
control  of  the  routing.  Because  information  of  flow  traffic  in  the  network  is  available,  the  algorithm  attempts  to 
achieve  intra-flow  interference  reduction  by  exploiting  node  mobility. 


Banerjee,  B.  (2003).  A  Self-Organizing  Auto-Associative  Network  for  the  Generalized  Physical  Design  of 
Microstrip  Patches.  IEEE  Transactions  on  Antennas  &  Propagation,  51(6),  1301-1306. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Hopfield-like  optimization  network,  information  extracting  self-organizing  neural  network  (IESONN),  patch 
antenna. 

Summary  abstract: 

The  current  work  deals  with  the  efficient  physical  design  of  patch  antennas  given  the  desired  parameters  like 
resonant  frequency  ,  feed  point  position  ,  substrate  thickness  ,  relative  permittivity  e  sub  v,  input  impedance  (Z  (=R 
+  jX  ),  and  efficiency  .  Based  loosely  on  the  analogy  of  perception  of  the  human  brain,  a  neurocomputing  network 
has  been  designed,  consisting  of  two  distinct  phases,  namely,  the  training  phase  and  the  application  phase.  The 
training  phase  accepts  as  input  the  exhaustive  set  of  the  said  parameters  for  patches  of  different  shapes  and  sizes  and 
determines  the  optimized  processors  (processors  that  adequately  define  the  information  topology  of  the  input  data 
set)  from  the  exhaustive  training  instances  using  a  set  of  information  extracting  self-organizing  neural  networks.  The 
outputs  of  the  training  phase  are  sets  of  processors,  being  the  number  of  different  shapes  of  patches  taken  into 
consideration.  The  application  phase  determines  the  shape  and  size  of  a  microstrip  antenna  when  its  desired 
parameters  are  presented  to  the  network  as  the  external  input.  This  is  achieved  by  comparing  the  external  input  with 
each  set  of  processors,  hence  determining  the  cost  due  to  each  comparison.  A  cost  matrix  is  thus  formed  which  when 
passed  through  an  optimization  network  gives  the  best  match  and  hence  the  shape  and  shape  determining  attributes 
of  the  patch  whose  parameters  had  been  passed  as  external  input. 


Banerjee,  B.  (2004).  Recognition  of  Partially  Occluded  Shapes  Using  a  Neural  Optimization  Network.  Machine 
Graphics  &  Vision  Journal,  Institute  of  Computer  Science  of  the  Polish  Academy  of  Sciences,  13(1/2),  3-23. 
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Summary  abstract: 
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The  current  work  presents  an  algorithm  for  recognition  of  partially  occluded  shapes  in  a  cluttered  scene.  The  imates 
are  represented  by  a  sequence  of  angles  subtended  at  the  corner  points.  The  cost  due  to  comparison  between  the 
input  cluttered  scene  and  the  stored  images  is  obtained  from  a  cost  function  designed  to  score  the  obtained 
information  in  the  form  of  a  cost  matrix  which  is  presented  to  the  input  of  an  optimization  network.  The  parameters 
of  the  optimization  network  are  determined  so  as  to  minimize  an  energy  function,  the  minima  of  which  occur  at  the 
solutions  of  the  problem.  The  results,  as  obtained  in  different  domains  (2D  shapes  and  projected  3D  shapes)  with 
different  degrees  of  occlusion,  provide  interesting  insights  into  the  operation  of  the  algorithm  as  well  as  avenues  for 
future  research. 


Banerjee,  B.,  &  Chandrasekaran,  B.  (2005).  Perceptual  and  Action  Routines  in  Diagrammatic  Reasoning  for  Entity- 
Reidentification.  In  In  Proceedings  of  24th  Army  Science  Conference.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  Objective  Force  requirements  of  responsiveness,  agility  and  versatility  call  for  digitized  graphical  decision 
support  interfaces  that  automate  or  otherwise  help  in  various  reasoning  tasks,  including  reasoning  with  visual  and 
diagrammatic  representations  that  are  ubiquitous  in  Army  situation  understanding,  planning  and  plan  monitoring.  In 
earlier  papers,  we  described  a  diagrammatic  reasoning  architecture,  and  demonstrated  the  approach  for  instances  of 
maneuver  recognition  and  information  fusion  for  entity-reidentification  problems.  The  current  paper  characterizes 
the  computational  properties  of  the  core  perception  and  path-finding  algorithms  that  are  an  important  part  of  the 
technology’s  application  for  a  class  of  fusion  problems.  This  analysis  is  important  since  the  practicality  of 
automating  diagrammatic  reasoning  for  Army  applications  depends  on  developing  algorithms  with  manageable 
complexity. 


Banerjee,  B.  (2006).  A  Layered  Abductive  Inference  Framework  for  Diagramming  Group  Motions.  Logic  Jnl  IGPL, 
Oxford  University  Press,  2006,  14(2):  363-378. 
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layered  abduction,  diagram,  group  motion. 

Summary  abstract: 

Many  events  in  different  domains  are  characterized  by  a  large  number  of  individual  moving  elements,  either  in 
pursuit  of  a  goal  in  groups  (as  in  military  operations),  or  subject  to  underlying  physical  forces  that  group  elements 
with  similar  motion  (as  in  weather  phenomena).  Visualizing  and  reasoning  about  happenings  in  such  domains  are 
often  facilitated  by  abstracting  the  mass  of  spatiotemporal  data  into  spatial  diagrams  of  group  motions,  and 
overlaying  them  on  abstractions  of  static  features,  like  maps.  The  standard  approach  has  been  to  use  a  clustering 
algorithm  to  group  the  entities  and  extract  their  motions,  which  often  produces  unsatisfactory  results  when  the  data 
is  noisy  and  incomplete.  In  this  paper,  we  present  for  the  task  a  multilayered  abductive  inference  framework  where 
hypotheses  largely  flow  upwards  from  raw  data  to  a  diagram,  but  there  is  also  a  top-down  control  that  asks  lower 
levels  to  supply  alternatives  if  the  higher  level  hypotheses  are  not  d""emed  sufficiently  coherent.  This  top-down- 
bottom-up  interplay  to  combine  numerical  clustering  algorithms  with  symbolic  knowledge  about  consistency  models 
the  flexibility  of  human  reasoning  in  "uch  tasks  that  enables  the  framework  to  successfully  deal  with  extremely 
noisy  and  incomplete  data.  We  present  experimental  results  as  obtained  by  deploying  the  proposed  framework  and 
discuss  several  issues  related  to  construction  of  such  diagrams,  from  data  sets  over  a  few  days  of  a  large  number  of 
military  units  engaged  in  exercises  at  the  National  Technical  Center,  California,  USA. 
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Banerjee,  B.  (2007).  String  Tightening  as  a  Self-Organizing  Phenomenon.  IEEE  Transactions  on  Neural  Networks, 
Vol  18,  No  5.,  18(5),  1463-1471. 
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Convex  hull,  homotopy,  neural  network, 
self-organization,  shortest  path,  smooth  path,  tighten  string. 

Summary  abstract: 

Abstract — The  phenomenon  of  self-organization  has  been  of  special  interest  to  the  neural  network  community 
throughout  the  last  couple  of  decades.  In  this  paper,  we  study  a  variant  of  the  self-organizing  map  (SOM)  that 
models  the  phenomenon  of  self-organization  of  the  particles  forming  a  string  when  the  string  is  tightened  from  one 
or  both  of  its  ends.  The  proposed  variant,  called  the  string  tightening  self-organizing  neural  network  (STON),  can  be 
used  to  solve  certain  practical  problems,  such  as  computation  of  shortest  homotopic  paths,  smoothing  paths  to  avoid 
sharp  turns,  computation  of  convex  hull,  etc.  These  problems  are  of  considerable  interest  in  computational  geometry, 
robotics  path-planning,  artificial  intelligence  (AI)  (diagrammatic  reasoning),  very  large  scale  integration  (VLSI) 
routing,  and  geographical  information  systems.  Given  a  set  of  obstacles  and  a  string  with  two  fixed  terminal  points 
in  a  2-D  space,  the  STON  model  continuously  tightens  the  given  string  until  the  unique  shortest  configuration  in 
terms  of  the  Euclidean  metric  is  reached.  The  STON  minimizes  the  total  length  of  a  string  on  convergence  by 
dynamically  creating  and  selecting  feature  vectors  in  a  competitive  manner.  Proof  of  correctness  of  this  anytime 
algorithm  and  experimental  results  obtained  by  its  deployment  have  been  presented  in  the  paper. 


Banerjee,  B.,  &  Chandrasekaran,  B.  (2006).  Constructing  Diagrams  Representing  Group  Motions.  In  A.  Blackwell, 
K.  Marriott,  &  A.  Shimojima  (Eds.),  Diagrammatic  Reasoning  and  Inference.  Lecture  Notes  in  Artificial 
Intelligence  (pp.  376-378).  2980,  Berlin:  Springer  Verlag. 
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Summary  abstract: 

Certain  domains,  such  as  military  activities  and  weather  phenomena,  are  characterized  by  a  large  number  of 
individual  elements  moving  in  a  field,  either  in  pursuit  of  an  organized  activity  in  groups  at  different  levels  of 
aggregation  (military  action),  or  subject  to  underlying  physical  forces  that  cluster  different  elements  in  different 
groups  with  common  motion  (weather).  Reasoning  about  phenomena  in  such  domains  is  often  facilitated  by 
abstracting  the  mass  of  data  into  diagrams  of  motions  of  groups,  and  overlaying  them  on  diagrams  that  abstract  static 
features  into  regions  and  curves.  Constructing  such  diagrams  of  motion  basically  calls  for  clustering  at  different  time 
instants  and  joining  the  centers  of  the  clusters  to  produce  lines  of  motion.  However,  because  of  incompleteness  and 
noisiness  of  data,  the  best  that  can  be  done  is  to  produce  plausible  hypotheses.  We  envision  a  multi-layered 
abductive  inference  approach  in  which  hypotheses  largely  flow  upwards  from  raw  data  to  a  diagram  to  be  used  by  a 
problem  solver,  but  there  is  also  a  top  -down  control  that  asks  lower  levels  to  supply  alternatives  if  the  original 
hypotheses  are  not  deemed  sufficiently  coherent. 


Banerjee,  B.,  &  Chandrasekaran,  B.  (2006).  A  Spatial  Constraint  Satisfaction  Framework  for  Synthesizing 
Perceptions  and  Actions  in  Diagrammatic  Reasoning.  In  In  Proceedings  of  1st  International  Workshop  on  Constraint 
Programming  for  Graphical  Applications.  In  conjunction  with  12th  International  Conference  on  Principles  and 
Practice  of  Constraint  Programming,  Nantes,  France. 
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ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Diagrammatic  reasoning  is  often  modeled  as  a  process  in  which  subtasks  may  be  solved,  as  appropriate,  either  by 
inference  from  symbolic  representations  or  by  information  extracted  by  perception  from  a  diagram,  and  additional 
subtasks  may  create  or  modify  objects  in  the  diagram.  The  required  perceptions  and  actions  are  defined  in  terms  of 
properties  and  relations  to  be  satisfied  by  a  set  of  diagrammatic  objects.  Performing  such  perceptions  and  actions  in 
general  requires  visual  problem  solving  for  humans.  This  paper  investigates  the  use  of  spatial  constraint  satisfaction 
for  automatically  synthesizing  solutions  for  per-  ceptions  and  actions.  Our  research  goal  is  to  develop  a  high-level 
language  that  is  finite,  extensible,  human-usable,  and  expressive  enough  to  describe  the  properties  of  desired 
perceptions  and  actions  as  constraints  specified  in  terms  of  well-defined  mathematical/logical  functions  and 
predefined  perceptions/actions;  to  compute  the  solutions  of  the  desired  perceptions/actions  and  diagrammatically 
represent  the  outcome  for  actions;  and  to  automatically  synthesize  their  programs,  thereby  transforming  them  into 
readily  executable  routines.  The  ideas  are  illustrated  by  several  examples  in  different  domains. 


Banerjee,  B.,  &  Chandrasekaran,  B.  (2006).  Synthesizing  Visual  and  Action  Routines  Using  Constraint 
Programming.  In  In  Proceedings  of  International  Conference  on  Diagrammatic  Representations  and  Reasoning  at 
Stanford  University. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

A  diagrammatic  problem-solver  requires  a  library  of  visual  routines  (VRs)  and  action  routines  (ARs)  -  the  VRs  are 
used  to  obtain  information  of  specified  types  from  the  diagram  and  ARs  to  modify  the  diagram  in  specified  ways. 

The  VRs/ARs  required  are  unbounded  -  a  new  domain  may  call  for  new  perceptions  and  actions.  We  report  on 
progress  on  our  research  in  building  an  automated  VR/AR  synthesis  system  that  would  take  as  input  the  definition  of 
a  new  routine  in  terms  of  existing  routines  in  the  library  and  well-defined  mathematical/logical  constraints  and 
synthesize  the  program  for  the  desired  routine  using  constraint  programming.  We  illustrate  the  ideas  by  means  of  an 
example. 


Banerjee,  B.,  &  Chandrasekaran,  B.  (2007).  A  Framework  for  Planning  Multiple  Paths  in  Free  Space.  In  In 
Proceedings  of  25th  Army  Science  Conference,  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Path  planning,  a  topic  of  much  interest  in  military  planning,  is  largely  treated  as  the  task  of  finding  the  best  path  with 
respect  to  some  criterion  such  as  length,  travel  time,  etc,  for  which  efficient  algorithms  are  already  available. 

Military  planning  requires  understanding  enemy  intentions  and  devising  unexpected  plans  to  fox  the  enemy  which 
calls  for  not  a  best  path,  but  a  number  of  alternative  paths.  We  study  the  problem  of  computing  multiple  paths  with 
different  properties,  such  as  all  paths  with  at  most  L  loops,  in  free  space  among  polygonal  obstacles  using  a 
framework  of  Voronoi  diagram.  The  complexity  of  the  algorithms  have  been  analyzed.  We  show  that  the  Voronoi 
diagram,  though  widely  used,  is  inadequate  to  represent  certain  important  properties  of  representative  paths  in  free 
space.  Further,  we  show  how  this  framework  might  be  applied  in  three  different  military  problems  -  entity 
reidentification,  ambush  analysis,  and  rapid  re-routing  in  urban  operations. 


Banerjee,  B.,  &  Chandrasekaran,  B.  (2007).  On  Automating  Perceptions  and  Actions  in  Reasoning  with  Military 
Diagrams.  In  In  Proceedings  of  25th  Army  Science  Conference,  Orlando,  FL. 
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Summary  abstract: 

Decision  support  systems  should  be  able  to  handle  representing  and  reasoning  with  diagrams,  since  they  are  a  major 
component  of  representations  in  army  situation  understanding  and  planning.  Diagrams  are  useful  because  desired 
information  about  spatial  properties  of  and  relations  among  diagrammatic  objects  can  be  perceived  from  a  diagram 
directly.  Spatially  relevant  changes  in  a  situation  or  plan  can  be  encoded  by  modifying  a  diagram  appropriately.  In 
this  paper,  we  present  an  approach  for  automating  perceptions  and  actions  on  a  diagram  to  assist  in  visual  reasoning 
needed  for  army  decision  making.  The  required  perceptions  and  actions  are  defined  in  terms  of  properties  and 
relations  to  be  satisfied  by  a  set  of  diagrammatic  objects.  This  paper  investigates  the  use  of  spatial  constraint 
satisfaction  for  automatically  synthesizing  solutions  for  perceptions  and  actions.  Our  research  goal  is  to  develop  a 
high-level  language  that  is  finite,  extensible,  humanusable,  and  expressive  enough  to  describe  the  properties  of 
desired  perceptions  and  actions  as  constraints  specified  in  terms  of  well-defined  mathematical/logical  functions  and 
predefined  perceptions/actions;  to  compute  the  solutions  of  the  desired  perceptions/actions  and  diagrammatically 
represent  the  outcome  for  actions;  and  to  automatically  synthesize  their  programs,  thereby  transforming  them  into 
readily  executable  routines.  The  ideas  are  illustrated  by  several  examples. 


Banerjee,  B.,  &  Chandrasekaran,  B.  (2007).  Representations  and  Strategies  for  Solving  Spatial  Problems  with 
Diagrams.  In  In  Proceedings  of  2007  IEEE  Symposium  on  Visual  Languages  and  Human-Centric  Computing,  Coeur 
d'AlSne,  Idaho. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Spatial  problems  (SP)  are  inevitable  in  reasoning  with  diagrams.  In  this  paper,  we  investigate  general  representations 
and  computational  strategies  for  a  SP-solver  such  that  it  can  accept  problems  from  a  human  in  a  high-level  language 
and  output  the  solution  without  human  intervention.  We  propose  a  language  in  which  a  variety  of 
domainindependent  2D  SPs  can  be  specified  in  terms  of  constraints.  The  constraints  are  specified  in  first-order  logic 
over  the  real  domain  using  a  vocabulary  of  objects,  properties,  relations  and  actions.  We  also  propose  two  general 
and  independent  computational  strategies  -  constraint  satisfaction  and  spatial  search  -  for  autonomously  solving  the 
SPs.  Ideas  about  how  to  make  these  strategies  computationally  efficient  are  discussed  and  illustrated  by  examples. 


Banerjee,  B.,  &  Chandrasekaran.,  B.  (2007).  Chapter  26.  A  Constraint  Satisfaction  Framework  for  Visual  Problem 
Solving.  In  F.  Benhamou,  N.  Jussien,  &  B.  O'Sullivan  (Eds.),  Trends  in  Constraint  Programming:  Hermes  Science. 
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Summary  abstract: 

AI  models  problem  solving  as  a  search  for  a  path  to  a  goal  state  from  an  initial  state.  Traditionally,  the  underlying 
representations  have  been  in  the  so-called  symbolic  framework,  i.e.  the  goal,  the  states  and  the  operators  that 
transform  the  states  are  all  represented,  as  compositions  of  predicate-symbolic  structures,  similar  to  natural 
language.  However,  in  real-world  problem  solving,  states  and  operators  often  have  perceptual  components,  such  as 
when  diagrams  are  used  in  problem  solving.  In  diagrammatic  reasoning  (DR),  problem  solving  proceeds 
opportunistically.  In  addition  to  the  traditional  symbolic  state  transformations,  perception  may  be  called  upon  to 
extract  certain  information  from  the  diagram,  or  actions  may  be  taken  on  the  diagram  to  create  or  modify  parts  of  it 
to  satisfy  a  given  description.  For  example,  a  problem  solving  episode  might  need  the  information  whether  there  is  a 
school  within  five  miles  of  the  agent's  home.  The  agent  has  to  solve  appropriate  visual  problems  -  abstract  his  home 
and  all  schools  in  a  given  map  as  points,  calculate  the  distances  of  schools  from  his  home,  and  compare  them  to  five 
miles  -  to  extract  the  required  information. 
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We  proposed  and  implemented  a  constraint  satisfaction  framework  for  solving  visual  problems  relevant  to  DR.  The 
specification  language  is  set-theoretic  with  constraints  specified  in  first  order  logic.  We  have  expressed  and  solved  a 
number  of  problems  including  those  that  require  constraint  solving  as  well  as  constraint  optimization,  with  discrete 
as  well  as  continuous  domains.  Though  most  of  the  problems  were  solved  efficiently,  there  were  two  problems  - 
grouping  and  computing  the  shortest  path  -  that  required  searching  through  a  combinatorially  large  set  of  solutions. 
We  also  illustrated  how  our  problem  solver  grows  smarter  by  solving  more  problems. 


Barnes,  M.,  Broome,  B.,  &  Chandrasekaran,  B.  (2003).  Knowledge  Environments  to  Support  Horizontal  Fusion.  In 
In  Proceedings  of  Army  Science  Conference.  Orlando,  FL. 
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Summary  abstract: 

While  a  great  deal  of  technology  is  in  place  to  support  the  tenets  of  DoD’s  Horizontal  Fusion  program,  considerable 
research  must  be  accomplished  to  achieve  the  program’s  goals  of  a  network  customized  to  its  users,  providing  both 
shared  situation  awareness  and  data  availability  that  is  limited  only  by  policy.  This  paper  outlines  some  of  the  issues 
that  must  be  addressed  to  make  that  vision  a  reality  and  some  of  the  current  Army  Research  Laboratory  initiatives  in 
place  to  address  them. 


Bames,  M.  J.,  McDermott,  P.,  Hutchins,  S.,  Gillan,  D.  J.,  &  Rothrock,  L.  (2003).  The  Presentation  of  Risk  and 
Uncertainty  in  the  Context  of  National  Missile  Defense  Simulations.  In  In  Proceedings  of  Collaborative  Technology 
Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003 
(pp.  197-200).  College  Park,  MD:  U.S.  Army  Research  Laboratory. 

Key  Words: 

Human  Performance 

Summary  abstract: 

Risk  perception  and  uncertainty  management  are  an  important  component  of  military  decision  making,  especially  in 
time-stressed  and  resource-limited  environments.  This  series  of  experiments  seeks  to  understand  how  to  present 
information  to  take  advantage  of  human  heuristics  while  minimizing  cognitive  biases.  We  have  taken  into  account 
research  on  uncertainty  perception,  framing,  mental  models,  and  perceptual  format.  The  result  is  a  first  experiment 
that  varies  the  format  of  information,  the  presentation  mode,  and  the  frame  of  information  to  determine  the  effect  on 
decision  making  and  situation  awareness  in  a  National  Missile  Defense  (NMD)  paradigm.  Subjects  had  to  decide 
how  to  allocate  a  reserve  supply  of  defensive  missiles  in  dynamic  NMD  scenarios  in  which  U.S.  cities  were 
attacked.  A  pilot  study  was  conducted  to  determine  which  graphical  format  best  supported  situation  awareness  and 
thus  would  be  used  in  the  simulation  study.  Results  from  the  pilot  study  indicated  that  a  "square"  representation 
better  supported  situation  awareness  with  only  a  minimal  time  increase.  Results  from  the  simulation  study  are 
pending  and  will  be  presented  in  the  poster. 


Bennett,  K.  B.,  Behymer,  K.,  Stansifer,  C.,  Shattuck,  L.,  Talcott,  C.,  &  Martinez,  S.  (2003).  Interface  Design  for 
Mobile  Army  Commanders.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society  Conference.  Denver,  CO.  (pp.  625-629). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  progress  in  developing  and  evaluating  a  prototype  interface  for  mobile,  forward-positioned  Army  commanders 
is  reported.  This  interface  currently  includes  novel  representations  of  friendly  and  enemy  combat  resources  and 
control  mechanisms  to  re-focus  granularity  of  attention.  Army  domain  experts  contributed  substantially  to  design 
and  evaluation  activities:  existing  and  proposed  interfaces  were  reviewed,  iterative  design  feedback  was  provided,  a 
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simulated  combat  scenario  was  designed,  and  three  empirical  evaluations  have  been  performed  with  Army  officers 
as  participants.  The  evaluations  have  compared  the  RAPTOR  interface  with  a  simulated  version  of  the  Army’s 
current  computerized  interface  (FBCB2);  the  RAPTOR  interface  has  generally  produced  better  performance.  The 
next  step  is  to  incorporate  graphical  representations  of  plans  into  the  interface.  These  representations  will  include 
information  regarding  time,  space,  objectives,  and  resources.  The  design  goal  is  to  facilitate  comparisons  between 
pre-planned  objectives  for  the  engagement  and  the  actual  progress  that  is  being  made.  Ultimately  these  graphical 
representations  should  facilitate  a  commander’s  capability  to  determine  when  current  progress  toward  goals  and 
resource  expenditures  have  deviated  (either  positively  or  negatively)  from  plan. 


Bennett,  K.  B.,  Posey,  S.  M.,  &  Shattuck,  L.  G.  (2008).  Ecological  Interface  Design  for  Military  Command  and 
Control.  Journal  of  Cognitive  Engineering  and  Decision  Making,  2(4.  Winter  2008),  349-385. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

The  authors  use  the  cognitive  systems  engineering  framework  to  design  and  evaluate  an  interface  for  military 
command  and  control.  They  discuss  analytic  tools  and  principles  of  this  framework  and  provide  concrete  examples 
(e.g.,  work  domain  analyses  for  U.S.  Army  tactical  operations  at  the  battalion  level).  They  also  discuss  principles  of 
ecological  interface  design,  including  direct  perception,  direct  manipulation,  and  the  perception-action  loop.  The 
translation  between  work  domain  analyses  and  the  specific  characteristics  of  the  interface  are  made  explicit.  The 
authors  describe  the  potential  for  this  interface  to  support  effective  decision  making  and  problem  solving,  including 
links  with  naturalistic  decision-making  approaches.  Evaluations  of  the  interface  have  been  positive  and  are  described 
briefly.  Actual  or  potential  applications  of  this  research  include  both  specific  interface  design  strategies  for  military 
command  and  control  and  general  interface  design  principles  for  this  category  of  work  domain. 


Bennett,  K.  B.,  &  Shattuck,  L.  (2005).  Advanced  Interfaces  to  Support  Continuous  Planning  and  Re-Planning 
During  Tactical  Operations.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

A  research  program  to  develop  decision  support  for  mobile  Army  commanders  has  resulted  in  a  prototype  interface. 
This  interface  was  developed  using  principles  of  Cognitive  Systems  Engineering,  and  initial  evaluations  were  very 
positive.  This  demonstration  showcases  recent  design  efforts  to  support  re-planning  during  tactical  operations,  which 
are  extremely  demanding  and  critical  for  mission  success.  Unfortunately,  these  activities  have  received  relatively 
little  attention  from  the  research  community.  The  prototype  interface  has  been  designed  to  assist  the  commander  in 
performing  these  activities.  This  interface  highlights  key  elements  of  the  battle  plan  (i.e.,  the  intended  course  of 
events),  the  battle  execution  (i.e.,  the  actual  course  of  events  as  they  unfold  over  time),  and  the  differences  between 
them. 


Bennett,  K.  B.,  &  Shattuck,  L.  G.  (2009).  Chapter  7.  Advancing  the  State-of-the-Art  in  Military  Command  & 
Control  Interfaces:  Ecological  Interface  Design.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision 
Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  II.  Presenting  Battlefield  Information  to 
Warfighters:  48hrBooks  by  Alion  Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research 
Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings 
from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final 
Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23, 
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2009.  1100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr- 
webtest. alionscience.com/AdaCta/ Agenda,  aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

The  discipline  of  human  factors  and  ergonomics  emerged  as  a  direct  result  of  problems  encountered  in  military 
systems  during  World  War  II.  Advances  in  technology  (e.g.,  radar  and  sonar)  created  new  capabilities  and 
opportunities;  it  also  created  the  potential  for  new  kinds  of  failures  (some  subtle,  some  spectacular).  Military 
contexts  continue  to  be  a  proving  ground  for  technological  innovation.  For  example,  a  wide  variety  of  new  forms  of 
data  have  become  available  for  today's  military  decision  makers  (e.g.,  fuel  and  ammunition  levels  in  individual 
vehicles,  sensor  data,  sattelite  and  UAV  imagery,  etc.).  Similarly,  advances  in  graphical  display  and  interface 
technology  (e.g.,  high-resolution,  bit-mapped  screens)  provide  new  options  in  the  presentation  of  these  data. 
Together,  these  new  technological  advances  provide  the  potential  to  improve  the  efficiency  and  effectiveness  of 
military  operations. 

Unfortunately,  this  potential  is  not  being  realized.  Existing  Army  systems  grossly  under-utilize  these  display  and 
interface  resources.  To  be  fair,  it  should  be  noted  that  it  is  not  a  problem  that  is  germane  to  just  the  Army;  the  same 
criticisms  can  be  leveled  at  many  other  work  domains  as  well.  The  general  question  is  how  can  we  leverage  the 
potential  that  is  provided  by  today's  hardware  and  software  technology  in  the  design  of  our  socio-technical  systems? 
More  specifically,  in  the  context  of  Army  command  and  control:  How  do  we  go  about  designing  interfaces  that 
commanders  will  find  both  powerful  and  easy  to  use? 

This  chapter  reviews  a  research  program  that  was  conducted  under  the  ARL  ADA  CTA  program  with  these 
questions  in  mind.  An  ecological  interface  was  designed  to  support  mobile  battalion-level  commanders  during 
tactical  operations.  The  particulars  of  this  interface,  the  framework  which  guided  its  development,  and  the 
implications  for  display  and  interface  design  in  general  will  be  discussed. 


Benvegnu,  E.,  Suri,  N.,  Hanna,  J.,  Combs,  V.,  R.,  W.,  &  Kovach,  J.  (2009).  Improving  Timeliness  and  Reliability  of 
Data  Delivery  in  Tactical  Wireless  Environments  with  Mockets  Communications  Library.  In  In  Proceedings  of  2009 
IEEE  Military  Communications  Conference  (MilCom  2009),  Boston,  MA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Network  centric  warfare  relies  on  the  timely  and  reliable  delivery  of  data  to  disparate  cooperating  nodes  in  tactical 
networking  environments.  Given  the  limited  bandwidth  available  and  the  unreliability  of  network  links,  data  often 
accumulates  in  application  and/or  network  queues,  resulting  in  increased  latency  in  the  delivery  of  the  data.  The 
Mockets  communications  library  addresses  this  problem  via  dynamic  message  replacement.  The  message 
replacement  functionality  of  Mockets  allows  the  system  to  drop  all  but  the  most  recent  message  within  a  specific 
message  flow  by  removing  older,  outdated  messages  from  the  queues.  This  paper  describes  and  evaluates,  in  the 
context  of  the  U.S.  Air  Force's  Joint  Battlespace  Infosphere  (JBI)  system,  the  timeliness  of  end-to-end  delivery  of 
data  using  the  Mockets  library. 

In  addition  to  dynamic  message  replacement,  other  capabilities  in  the  Mockets  library  include  options  for  reliable  vs. 
unreliable  and  sequenced  vs.  unsequenced  delivery  of  data,  detailed  statistics  and  feedback  regarding  the  connection, 
and  assignment  and  dynamic  adjustment  of  priorities  of  messages.  This  paper  provides  a  qualitative  analysis  of  these 
different  capabilities  and  their  suitability  to  address  the  transport  requirements  in  JBI.  It  also  provides  a  quantitative 
comparison  of  Mockets  with  SCTP  and  SCPS-TP,  which  are  similar  technologies  with  existing  available  candidate 
implementations.  Our  results  show  that  the  Mockets  library  with  the  message  replacement  significantly  outperforms 
these  other  transport  protocols. 


Bharathan,  V.,  &  Josephson,  J.  R.  (2006).  An  Abductive  Framework  for  Level  One  Information  Fusion,.  In  In 
Proceedings  of  Fusion  2006:  The  9th  International  Conference  on  Information  Fusion,  Florence  (Italy). 


82 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 


Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Artificial  Intelligence,  Tracking  and  Surveil¬ 
lance,  Fusion  Architecture,  C4ISR,  Situation  Assessment. 

Summary  abstract: 

This  article  argues  for,  and  describes  some  of  the  advantages  of,  construing  Level  One  Information  Fusion,  as  a  task 
of  Abductive  Infer-  ence  or  Inference  to  the  Best  Explanation.  Such  an  approach  enables  certain  benefits,  such  as,  an 
expectation-based  critique  of  hypotheses,  and  an  ele-  gant  system  for  revising  old  beliefs,  which  may  gain-  fully  be 
exploited.  It  also  introduces  several  relevant  dimensions  to  reasoning  based  on  the  exlanatory  re-  lations  between 
hypotheses  and  data,  that  are  closed  to  traditional  approaches.  The  design  principles  of  a  software  system,  Smart- 
ASAS,  that  attempts  to  solve  the  Level  One  Fusion  task  of  entity  tracking  and  re-identification,  are  described,  along 
with  an  example  that  illustrates  its  capabilities. 


Bharathan,  V.,  &  Josephson,  J.  R.  (2006).  Belief  Revision  Controlled  by  Meta-Abduction.  Logic  Jnl  IGPL,  Oxford 
University  Press,  14(2),  271-285. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

We  describe  an  abductive  reasoning  agent  that  is  able  to  change  its  mind  and  appropriately  revise  previous 
conclusions  when  it  encounters  a  reasoning  dilemma.  Abductive  inference  (by  which  we  mean  "Inference  to  the 
Best  Explanation")  is  an  ampliative  inferential  process;  that  is,  the  conclusion  goes  beyond  merely  extracting 
information  already  present  in  the  premises,  perhaps  unobviously.  Thus,  abductive  inference  is  inherently  fallible, 
and  so  it  is  desirable  for  a  reasoning  agent  relying  on  abduction  to  be  able  to  correct  mistaken  previous  conclusions. 
A  number  of  frameworks  have  been  proposed  for  how  to  model  and  implement  the  logic  of  how  an  agent  might 
change  its  mind:  the  non-monotonic  reasoning  techniques  of  default  reasoning,  Justification  (and  Assumption)-based 
truth  maintenance  systems  and  AGM  theory,  are  prominent  examples.  These  approaches,  whatever  their  differences, 
share  the  property  that  they  all  seek  universal  solutions  to  the  problem  of  belief  revision.  In  contrast,  we  propose  to 
take  advantage  of  the  specific  structure  of  abductive  reasoning  to  identify  revision  candidates  among  earlier  beliefs, 
to  propose  specific  revisions,  to  select  among  possible  revisions,  and  to  make  the  requisite  changes  to  the  system  of 
beliefs.  These  adjustments  are  performed  through  meta-abductive  processing  over  the  recorded  steps  in  an  abductive 
agent's  reasoning  trace. 


Bharathan,  V.,  &  Josephson,  J.  R.  (2007).  Detecting  and  Correcting  Mistakes  in  Information  Fusion.  In  In 
Proceedings  of  25th  Army  Science  Conference,  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  article  scrutinizes  one  inherent  pitfall  in  automated  information  fusion  -  its  fallibility  as  an  artifact  of  dealing 
with  the  real  world.  Since  it  is  highly  desirable  to  avoid  contamination  of  further  processing  by  past  mistakes,  we 
investigate  the  nature  of  the  recovery  process  in  a  prototype  agent  that  performs  the  Level  One  Information  Fusion 
task  of  entity  tracking  and  re-identification.  Smart-  ASAS.  Smart-ASAS  attempts  to  solve  this  fusion  task  by 
treating  it  as  one  of  Abductive  Inference  or  Inference  to  the  Best  Explanation.  We  discover  that  the  problem  space 
bounding  recovery  from  errors  is  exponential  in  nature,  but  investigate  the  possibility  of  handling  this 
computationally  complex  problem  with  some  proposed  heuristics  that  would  result  in  some  satisficing  solution. 
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Binder,  W.,  &  Suri,  N.  (2009).  Green  Computing:  Energy  Consumption  Optimized  Service  Hosting.  In  In 
Proceedings  of  35th  International  Conference  on  Current  Trends  in  Theory  and  Practice  of  Computer  Science 
(SOFSEM-2009),  Czech  Republic.  Lecture  Notes  in  Computer  Science  (LNCS),  Volume  5404,  pp.  1 17-128. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Green  computing,  energy  conservation,  energy  efficiency,  data  centers,  service  hosting,  service  dispatch,  service- 
level  agreements 

Summary  abstract: 

Green  Computing  is  a  recent  trend  towards  designing,  building,  and  operating  computer  systems  to  be  energy 
efficient.  While  programs  such  as  Energy  Star  have  been  around  since  the  early  1990s,  recent  concerns  regarding 
global  climate  change  and  the  energy  crisis  have  led  to  renewed  interest  in  Green  Computing.  Data  centers  are  a 
significant  consumers  of  energy  -  both  to  power  the  computers  as  well  as  to  provide  the  necessary  cooling.  This 
paper  proposes  a  new  approach  to  reduce  energy  utilization  in  data  centers.  In  particular,  our  approach  relies  on 
consolidating  services  dynamically  onto  a  subset  of  the  available  servers  and  temporarily  shutting  down  servers  in 
order  to  conserve  energy.  We  present  initial  work  on  a  probabilistic  service  dispatch  algorithm  that  aims  at 
minimizing  the  number  of  running  servers  such  that  they  suffice  for  meeting  the  quality  of  service  required  by 
service-level  agreements.  Given  the  estimated  energy  consumption  and  projected  growth  in  data  centers,  the 
proposed  effort  has  the  potential  to  positively  impact  energy  consumption. 


Bolstad,  C.  (2005).  Measuring  Distributed  Situation  Awareness.  In  In  Proceedings  of  Cognitive  Performance  05 
workshop,  Las  Vegas,  NV. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

Distributed  Teams 

•  Predominant  structure  for  command  and  control  operations 

•  Need  new  methods  analyze  and  assess  team  performance 

•  Difficulties  For  Distributed  Teams 
§  Rapidly  Changing  Environment 
§  Large  Amounts  of  Data 

§  Sharing  Information  Between  and  Across  Units 

•  Distributed  Teams  Need  High  SA 

Definition  of  SA 

Situation  Awareness  is  the  Perception  of  the  Elements  in  the  Environment  within  a  Volume  of  Time  and  Space,  the 
Comprehension  of  their  Meaning,  and  the  Projection  of  their  Status  in  the  Near  Future.  (Endsley,  1988) 


Bolstad,  C.,  Schneider,  M.,  Graham,  J.,  &  Gonzalez,  C.  (2004).  Collaborative  Tools  and  Shared  Mental  Models.  In 
In  Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  reports  on  an  experiment  conducted  to  measure  collaboration  tool  usage  and  shared  mental  models  in 
Army  Command  and  Control.  Several  teams  of  soldiers  participated  in  a  two-week  simulation  experiment.  During 
the  simulation,  the  run  was  stopped  and  participants  completed  a  5-minuted  on-line  questionnaire  in  which  they 
were  asked  to  rate  their  workload  and  other  teammates  workload  as  well  as  provide  information  on  the  collaboration 
tools  used  during  the  last  hour  of  the  exercise.  Shared  mental  models  was  measured  using  congruency  between  each 
person’s  rating  of  their  own  workload  and  their  teammates  perception  of  their  workload.  Overall,  using  the  same 
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collaboration  tool,  irrespective  of  the  tool  type,  to  communicate  between  team  members  lead  to  higher  shared  mental 
models. 


Bolstad,  C.  A.,  Cuevas,  H.  M.,  Costello,  A.  M.,  &  Rousey,  J.  (2005).  Improving  Situation  Awareness  through  Cross- 
Training.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  study  investigated  how  cross-training,  particularly  in  a  leadership  role,  may  assist  individuals  in  better 
understanding  the  task  requirements  of  their  fellow  team  members,  and,  thereby,  increase  their  shared  situation 
awareness.  Data  was  collected  from  a  training  exercise  at  the  Joint  Personnel  Recovery  Agency.  Participants  were 
assigned  to  one  of  4  teams  (Navy,  Army,  Special  Ops,  or  Joint  Service)  and  completed  a  simulated  exercise 
designed  to  mimic  real  life  events  in  a  recovery  center.  Each  player  was  rotated  though  the  various  positions  and 
teams  such  that  everyone  had  a  chance  to  be  a  team  director  (lead  person)  and  a  team  member  in  each  of  the  4 
teams.  Situation  awareness  was  measured  during  the  exercise  using  the  SAGAT  technique.  Overall,  results  suggest 
that  cross-training  may  lead  to  improved  situation  awareness.  Participants,  on  average,  exhibited  greater  situation 
awareness  following  experience  in  the  director  role  than  prior  to  director  experience. 


Bolstad,  C.  A.,  Cuevas,  H.  M.,  Gonzalez,  C.,  &  Schneider,  M.  (2005).  Modeling  Shared  Situation  Awareness.  In  In 
Proceedings  of  14th  Conference  on  Behavior  Representation  in  Modeling  and  Simulation  (BRIMS),  Universal  City, 
CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Situation  Awareness,  Social  Network  Analysis,  Distributed  Team  Performance 

Summary  abstract: 

This  study  presents  an  initial  computational  model  of  shared  situation  awareness  (SA)  based  upon  data  collected 
from  a  simulated  training  exercise,  designed  to  mimic  real  life  events  in  a  military  personnel  recovery  center. 
Situation  awareness  was  measured  during  the  exercise  using  the  Situation  Awareness  Global  Assessment  Technique 
(SAGAT).  Our  initial  model  examined  how  well  five  factors  (social  network  distance,  physical  distance,  rank 
similarity,  branch  similarity,  and  experience  similarity)  predicted  shared  SA.  Overall,  regression  analyses 
highlighted  the  significant  influence  of  geographical  distribution  (physical  distance)  on  the  development  of  shared 
SA  and  frequency  of  communications  amongst  team  members.  The  discussion  centers  on  the  need  for  developing 
useful  technological  tools  and  techniques  for  supporting  communication  and  collaboration  among  distributed  teams. 


Bolstad,  C.  A.,  &  Endsley,  M.  R.  (2003).  Measuring  Shared  and  Team  Situation  Awareness  in  the  Army's  Future 
Objective  Force.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society 
Conference.  Denver,  CO.  (pp.  369-373). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

A  simulation  exercise  was  conducted  to  assess  the  effectiveness  of  a  new  Army  force  structure  called  Objective 
Force.  This  paper  will  describe  how  shared  and  team  situation  awareness  (SA)  were  measured  and  analyzed  in  this 
experiment  with  regards  to  how  well  the  new  force  structure  supports  both  SA  at  the  individual  officer  level,  within 
teams  and  across  teams.  Shared  and  Team  SA  were  measured  using  the  Situation  Awareness  Global  Assessment 
Technique  (SAGAT). 
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Bolstad,  C.  A.,  &  Endsley,  M.  R.  (2003).  Tools  for  Supporting  Distributed  Team  Collaboration.  In  In  Proceedings 
Of  The  Human  Factors  And  Ergonomics  Society  47th  Annual  Meeting — 2003  Human  Factors  and  Ergonomics 
Association  (HFEA). 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Collaboration  plays  an  important  role  in  team  tasks.  Teams  are  also  functioning  more  often  in  a  distributed  fashion. 
If  individuals  are  to  work  efficiently  in  a  distributed  fashion  they  will  need  collaborative  tools  and  systems  to 
exchange  information  and  most  importantly  Situation  Awareness  (SA).  Little  guidance  exists  as  to  which  tools  are 
appropriate  for  coiiaborative  tasks  or  situations.  The  present  paper  presents  a  taxonomy  of  collaboration  and,  based 
on  this  taxonomy,  information  is  provided  on  which  classes  of  collaborative  tools  and  techniques  are  most  useful  for 
different  types  of  tasks  and  situations. 


Bolstad,  C.  A.,  &  Endsley,  M.  R.  (2003).  Tools  for  Supporting  Team  Collaboration.  In  In  Proceedings  of  47th 
Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Collaboration  plays  an  important  role  in  team  tasks.  Teams  are  also  functioning  more  often  in  a  distributed  fashion. 
If  individuals  are  to  work  efficiently  in  a  distributed  fashion  they  will  need  collaborative  tools  and  systems  to 
exchange  information  and  most  importantly  Situation  Awareness  (SA).  Little  guidance  exists  as  to  which  tools  are 
appropriate  for  coiiaborative  tasks  or  situations.  The  present  paper  presents  a  taxonomy  of  collaboration  and,  based 
on  this  taxonomy,  information  is  provided  on  which  classes  of  collaborative  tools  and  techniques  are  most  useful  for 
different  types  of  tasks  and  situations. 


Bolstad,  C.  A.,  &  Endsley,  M.  R.  (2003).  Tools  for  Supporting  Team  Situational  Awareness  (SA)  and  Collaboration 
in  Army  Operations.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and 
Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University 
College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

Situation  awareness  (SA)  plays  an  important  role  in  Army  operations.  Within  the  future  Objective  Force, 
considerable  effort  is  being  directed  at  developing  tools  to  improve  SA  for  individual  soldiers  and  for  supporting 
shared  SA  and  collaboration  across  teams.  Little  guidance  exists  as  to  which  tools  are  appropriate  for  collaborative 
tasks  or  situations.  The  present  paper  presents  a  taxonomy  of  collaboration  and,  based  on  this  taxonomy,  information 
is  provided  on  which  classes  of  collaborative  tools  and  techniques  are  most  useful  to  the  Army  for  different  types  of 
tasks  and  situations. 


Bolstad,  C.  A.,  &  Endsley,  M.  R.  (2004).  Collaboration  Tools  for  the  Army’s  Future  Force.  In  In  Proceedings  of  the 
Human  Performance,  Situation  Awareness  and  Automation:  Issues  and  Considerations  for  the  21st  Century 
Conference,  Daytona  Beach,  FL. 

Key  Words: 
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ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Collaboration  Tools,  Team  Performance,  Situation  Awareness,  Army 

Summary  abstract: 

Collaboration  tools  are  used  to  facilitate  the  communication  and  exchange  of  information  among  team  members  who 
are  working  together  to  complete  a  shared  task.  The  Army  has  under  development  a  wide  variety  of  tools  for 
supporting  collaboration.  In  addition,  many  commercial  products  have  also  been  developed  for  supporting 
collaborative  activities.  Each  of  these  tools  offers  very  different  types  of  capabilities,  however,  and  little  guidance 
exists  as  to  which  tools  provide  the  highest  levels  of  situation  awareness  (SA)  and  are  appropriate  for  which  types  of 
collaborative  tasks  or  situations  relevant  to  Army  Operations.  An  evaluation  of  these  tools  for  Army  command  and 
control  operations  was  conducted  in  a  simulation  exercise  at  Ft.  Leavenworth.  Overall,  soldiers  found  to  that  faceto- 
face  communication,  a  domain  mapping  tool  and  instant  messaging  were  most  effective  for  their  tasks. 


Bolstad,  C.  A.,  &  Endsley,  M.  R.  (2004).  Collaboration  Tools  for  the  Army's  Future  Force.  In  In  Proceedings  of 
Human  Performance,  Situation  Awareness  and  Automation:  Issues  and  Considerations  for  the  21st  Century. 

Daytona  Beach,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Collaboration  Tools,  Team  Performance,  Situation  Awareness,  Army 

Summary  abstract: 

Collaboration  tools  are  used  to  facilitate  the  communication  and  exchange  of  information  among  team  members  who 
are  working  together  to  complete  a  shared  task.  The  Army  has  under  development  a  wide  variety  of  tools  for 
supporting  collaboration.  In  addition,  many  commercial  products  have  also  been  developed  for  supporting 
collaborative  activities.  Each  of  these  tools  offers  very  different  types  of  capabilities,  however,  and  little  guidance 
exists  as  to  which  tools  provide  the  highest  levels  of  situation  awareness  (SA)  and  are  appropriate  for  which  types  of 
collaborative  tasks  or  situations  relevant  to  Army  Operations.  An  evaluation  of  these  tools  for  Army  command  and 
control  operations  was  conducted  in  a  simulation  exercise  at  Ft.  Leavenworth.  Overall,  soldiers  found  to  that  faceto- 
face  communication,  a  domain  mapping  tool  and  instant  messaging  were  most  effective  for  their  tasks. 


Bolstad,  C.  A.,  &  Endsley,  M.  R.  (2005).  Measurement  of  Situation  Awareness  in  Military  Operations.  In  In 
Proceedings  of  Cognitive  Performance  05  workshop,  Las  Vegas,  NV. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

Goals  of  SA  Measurement 

•  Investigations  of  SA  in  Military  Operations 

-  What  are  the  critical  skills/abilities  that  lead  to  high  SA? 

-  What  factors  hinder  SA  the  most? 

-  How  do  soldiers  maintain  SA  under  harsh  operational  conditions? 

-  What  strategies  lead  to  high  SA? 

-  How  does  SA  develop  within  and  between  teams? 

•  Evaluation  of  System  Designs 

-  Do  new  technologies  actually  improve  SA? 

-  Which  aspects  of  SA  are  hurt  by  technology? 

•  Evaluation  of  Training  Programs 

-  How  effective  are  new  training  techniques? 


Bolstad,  C.  A.,  &  Endsley,  M.  R.  (2006).  Choosing  Team  Collaboration  Tools:  Lessons  from  Disaster  Recovery 
Efforts.  Ergonomics  in  Design,  13(4),  7-14. 
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Journal  Article 

Summary  abstract: 

Extraodinary  situations,  such  as  natural  disasters,  teach  us  the  importance  of  matching  communication  support  tools 
to  the  task. 

ON  DECEMBER  26,  2004,  THE  MOST  powerful  earthquake  in  more  than  40  years  struck  deep  under  the  Indian 
Ocean,  triggering  massive  tsunamis  that  obliterated  cities,  seaside  communities,  and  holiday  resorts.  It  killed  more 
than  150,000  people  in  a  dozen  countries  (CNN,  2004).  A  complex  coordination  and  collaboration  effort  between 
relief  organizations  and  government  agencies  was  started  to  deliver  aid  to  the  survivors.  At  the  time  of  this  writing, 
25  developed  countries  and  79  major  international  organizations  had  provided  aid  for  this  effort  (CNN,  2005bj 
Nationmaster,  2005).  However,  collaboration  among  these  organizations  did  not  go  smoothly. 

This  massive  relief  operation  brought  food  and  medicine  to  most  large  population  groups  but,  nearly  a  month  after 
the  tragedy,  continued  to  be  hampered  by  insufficient  coordination,  according  to  a  draft  report  by  Rob  Holden, 
worldwide  operations  manager  for  the  World  Health  Organization  (Nakashima,  2005).  Although  the  sheer 
complexity  of  this  operation  overshadows  most  team  collaboration  efforts,  it  points  out  the  need  for  useful 
collaboration  tools  and  techniques  to  aid  these  and  other  organizations'  interactions. 

In  this  article,  we  present  examples  of  coordination  issues  associated  with  a  major  natural  disaster  and  how 
collaboration  tools  are  being  used  to  support  the  coordinated  relief  efforts. 


Bolstad,  C.  A.,  Riley,  J.  M.,  Jones,  D.  G.,  &  Endsley,  M.  R.  (2002).  Using  Goal  Directed  Task  Analysis  with  Army 
Brigade  Officers.  In  In  Proceedings  of  46th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society. 
Baltimore,  MD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

A  greater  understanding  of  team  cognitive  processes  can  be  facilitated  by  identifying  the  individual  goals  of  the 
team  members  and  their  situation  awareness  (SA)  requirements.  In  some  environments,  such  as  military  operations, 
the  shear  complexity,  size,  and  composition  of  the  team  make  this  research  quite  challenging.  Using  a  form  of 
cognitive  task  analysis,  we  have  developed  an  approach  to  address  some  of  these  team  issues.  In  this  paper  we 
discuss  the  use  of  goal  directed  cognitive  task  analysis  (GDTA)  to  obtain  an  accurate  depiction  of  the  SA 
requirements  and  key  goals  for  several  brigade  officers.  We  further  discuss  how  this  information  is  being  used  to 
address  team  issues  such  as  designing  systems  for  enhancing  team  performance  and  decision  making  with  Army 
brigade  officers. 


Bolstad,  C.  A.,  Wright,  M.,  Endsley,  M.  R.,  &  Jones,  D.  G.  (2003).  Measurement  of  Shared  Situation  Awareness  in 
the  Future  Objective  Force.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and 
Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University 
College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

A  simulation  exercise  was  conducted  at  the  end  of  January  at  the  Battle  Command  Battle  Lab  at  Fort  Leavenworth 
with  the  Objective  Force  Unit  of  Action  (UA)  structure.  This  paper  will  describe  how  situation  awareness  (SA)  was 
measured  in  this  experiment  and  will  describe  how  the  data  will  be  analyzed  with  regards  to  how  well  the  new  force 
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structure  supports  both  SA  at  the  individual  officer  level,  within  teams  (e.g.,  within  a  UA  cell),  and  across  teams 
(e.g.,  between  UA  cells). 


Bower,  J.  I.  (2004).  The  Impact  of  Multimedia  Communications  on  Understanding  and  Recall:  An  Empirical  Study. 
In  In  Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  research  focused  on  the  design  of  software  environments  to  support  rich  asynchronous  communication  and  to 
understand  the  impact  of  such  communications  on  understanding  and  recall  in  a  military  context.  The  goal  was  to 
explore  how  the  design  of  a  multimedia  asynchronous  communication  tool  influenced  the  understanding  and  recall 
of  information.  46  ROTC  cadets  each  received  a  company  Operations  Order,  and  based  on  that  were  required  to 
write  a  platoon  OPORD  followed  by  two  recall  questions.  Half  of  the  cadets  received  the  company  OPORD  in  a 
conventional  text  format  that  included  a  static  map  overlay.  The  other  half  received  the  OPORD  with  4  subsections 
presented  with  a  multimedia  presentation  tool  that  included  synchronized  animation  and  voice  narration.  Overall, 
our  findings  show  that  the  group  that  viewed  the  multimedia  presentation  recalled  mission  critical  information  26- 
47%  more  often. 


Boy,  G.,  &  Bradshaw,  J.  M.  (2004).  Interaction  Design  of  Highly  Automated  Domain-Specific  Systems.  In  In 
Proceedings  of  ACM  SIGCHI  Conference  on  Human  Factors  in  Computing  Systems  (CHI  2004)  World  Computer 
Conference  (WCC).  Vienna,  Austria. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  main  learning  objectives  are  to  introduce  practitioners  and  researchers  to  emerging  changes  in  the  way  people 
interact  with  machines,  and  in  particular  to  the  shift  from  direct  manipulation  to  agent  management  in  specific 
domains  such  as  aviation,  space  exploration,  military,  nuclear  industry,  automobile  industry,  telecommunications, 
and  medicine.  By  the  end  of  the  tutorial,  participants  will  be  able  to  better  understand  and  more  effectively  use 
current  concepts  in  interaction  design  of  highly  automated  domain-specific  systems. 

The  major  features  of  this  tutorial  are:  an  introduction  to  similarities  and  differences  between  human-centered  and 
technology-centered  approaches  to  interaction  design  of  highly  automated  domain-specific  systems;  a  development 
of  the  concept  of  cognitive  function  as  a  common  entity  that  is  useful  for  the  representation  of  both  human  and 
software  agents  (i.e.,  automation);  a  presentation  of  the  necessary  cognitive  science  knowledge  within  the  scope  of 
the  currently  emerging  industrial  agent  technology;  a  presentation  of  the  tradeoffs  between  direct  manipulation  and 
agent  management;  live  and  video  demonstrations  of  how  agents  can  be  used  to  facilitate  the  communication, 
cooperation  and  coordination  between  various  activities  that  include  training  and  operations. 

The  lecturers  will  invite  participants  to  interact  as  much  as  possible  providing  to  the  overall  group  their  questions 
and  problems  from  their  own  domain  and  situation.  Some  exercises  will  be  devoted  to  solving  some  of  these 
questions  in  a  very  interactive  way.  In  addition,  the  GEM  pRoom  software  will  be  used  to  carry  out  effective 
participatory  design  of  human-centered  automation.  Depending  on  the  number  of  attendees,  the  sessions  may  be 
differently  organized.  The  instructors  will  need  to  know  the  approximate  number  of  attendees  a  while  in  advance  to 
organize  the  final  setup  of  the  course. 


Bradshaw,  J.,  Uszok,  A.,  Jeffers,  R.,  Suri,  N.,  Hayes,  P.,  Burstein,  M.,  Acquisti,  A.,  Benyo,  B.,  Breedy,  M., 
Carvalho,  M.,  Diller,  D.,  Johnson,  M.,  Kulkami,  S.,  Lott,  J.,  Sierhuis,  M.,  &  Van  Hoof,  R.  (2003).  Representation 
and  Reasoning  for  DAML-Based  Policy  and  Domain  Services  in  KAoS  and  Nomads.  In  In  Proceedings  of  2nd 
International  Joint  Conference  on  Autonomous  Agents  and  Multi-Agent  Systems  (AAMAS  2003).  Melbourne, 
Australia. 
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Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

social  order,  conventions,  norms,  social  control;  cultural 

norms  and  institutions,  ontologies  for  agents  and  social 

modeling;  ontologies  in  agent-based  information  systems  and 

knowledge  management,  DAML,  policy,  domains,  KAoS, 

Nomads,  human-agent  teamwork,  adjustable  autonomy, 
coalition,  augmented  cognition,  cognitive  prosthesis 

Summary  abstract: 

To  increase  the  assurance  with  which  agents  can  be  deployed  in  operational  settings,  we  have  been  developing  the 
KAoS  policy  and  domain  services.  In  conjunction  with  Nomads  strong  mobility  and  safe  execution  features,  KAoS 
services  and  tools  allow  for  the  specification,  management,  conflict  resolution,  and  enforcement  of  DAML-based 
policies  within  the  specific  contexts  established  by  complex  organizational  structures.  In  this  paper,  we  will  discuss 
results,  issues,  and  lessons  learned  in  the  development  of  these  representations,  tools,  and  services  and  their  use  in 
military  and  space  applications 


Bradshaw,  J.  M.,  Jung,  H.,  Kulkarni,  S.,  &  Taysom,  W.  (2004).  Dimensions  of  Adjustable  Autonomy  and  Mixed- 
Initiative  Interaction.  In  M.  Nickles,  M.  Rovatsos,  &  G.  Weiss  (Eds.),  Agents  and  Computational  Autonomy: 
Potential,  Risks,  and  Solutions.  Lecture  Notes  in  Computer  Science.  Also  Presented  At:  Presentation  at  the 
University  of  Rochester.  Rochester,  Ny  (Vol.  2969,  pp.  17-39).  Berlin:  Springer  Verlag. 

Key  Words: 
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Book  Chapter/Section 

Summary  abstract: 

Several  research  groups  have  grappled  with  the  problem  of  characterizing  and  developing  practical  approaches  for 
implementing  adjustable  autonomy  and  mixed-initiative  interaction  in  deployed  systems.  However,  each  group  takes 
a  little  different  approach  and  uses  variations  of  the  same  terminology  in  a  somewhat  different  fashion.  In  this 
chapter,  we  will  describe  some  common  dimensions  in  order  to  better  understand  these  important  but  ill- 
characterized  topics.  We  will  also  sketch  the  approach  to  implementation  we  are  developing  in  the  context  of  our 
research  on  policygoverned  autonomous  systems. 


Bradshaw,  J.  M.,  Feltovich,  P.  J.,  Jung,  H.,  Kulkarni,  S.,  Allen,  J.,  Bunch,  L.,  Chambers,  N.,  Galescu,  L.,  Jeffers,  R., 
Johnson,  M.,  Sierhuis,  M.,  Taysom,  W.,  Uszok,  A.,  &  Van  Hoof,  R.  (2005).  Policy-Based  Coordination  in  Joint 
Human-Agent  Activity.  In  In  Proceedings  of  IEEE  International  Conference  on  Systems,  Man,  and  Cybernetics.  The 
Hague,  The  Netherlands. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Policy,  KAoS  coordination,  joint  activity,  human-agent  interaction 

Summary  abstract: 

In  this  paper,  we  outline  an  approach  to  policy  based  coordination  in  joint  human-agent  activity.  The  approach  is 
grounded  in  a  theory  of  joint  activity  originally  developed  in  the  context  of  discourse,  and  now  applied  to  the 
broader  realm  of  human  agent  interaction.  We  have  been  gradually  implementing  selected  aspects  of  policy  based 
coordination  within  the  KAoS  services  framework  and  have  been  developing  a  body  of  examples  that  will  guide 
additional  testing  of  these  ideas  through  detailed  studies  of  work  practice. 
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Bradshaw,  J.  M.  (2008).  How  to  Do  with  Owl  What  People  Say  You  Can't.  In  In  Proceedings  of  2008  IEEE 
Conference  on  Policy,  Palisades,  NY. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

WHAT  IS  OWL? 

(£  OWL  stands  for  Web  Ontology  Language 

f  OWL  is  built  on  top  of  RDF  and  written  in  XML 

</:  OWL  was  designed  to  be  interpreted  by  computers,  not  people 

f  OWL  has  three  sublanguages:  OWL-Full,  OWLDL,  and  OWL -Lite 

(£  OWL  is  a  Web  standard 

f  The  use  of  OWL  is  not  restricted  to  Web  applications 


Bradshaw,  J.  M.,  Acquisti,  A.,  Allen,  J.,  Breedy,  M.,  Bunch,  L.,  Chambers,  N.,  Galescu,  L.,  Goodrich,  M.,  Jeffers, 
R.,  Johnson,  M.,  Jung,  H.,  Kulkami,  S.,  Lott,  J.,  Olsen,  D.,  Sierhuis,  M.,  Suri,  N.,  Taysom,  W.,  Tonti,  G.,  Uszok,  A., 
&  van-Hoof,  R.  (2004).  Teamwork-Centered  Autonomy  for  Extended  Human-Agent  Interaction  in  Space 
Applications.  In  In  Proceedings  of  AAAI  Spring  Symposium.  Stanford,  CA. 
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Summary  abstract: 

This  paper  summarizes  our  efforts  to  bring  together  and  extend  the  best  in  current  theory  and  technologies  for 
teamwork-centered  autonomy  for  space  applications.  Traditional  planning  technologies  at  the  foundation  of 
intelligent  robotic  systems  typically  take  an  autonomycentered  approach,  with  representations,  mechanisms,  and 
algorithms  that  have  been  designed  to  ingest  a  set  of  goals  and  output  a  complete  plan  in  the  most  efficient  and 
sound  fashion  possible.  A  teamwork-centered  autonomy  approach,  on  the  other  hand,  takes  as  a  beginning  premise 
that  people  are  working  in  parallel  alongside  autonomous  systems,  and  hence  adopts  the  stance  that  the  processes  of 
understanding,  problem  solving,  and  task  execution  are  necessarily  incremental,  subject  to  negotiation,  and  forever 
tentative.  Thus,  a  successful  approach  to  teamwork-centered  autonomy  will  require  that  every  element  of  the 
autonomous  system  be  designed  to  facilitate  the  kind  of  give-and-take  that  quintessentially  characterizes  natural  and 
effective  teamwork  among  groups  of  people.  We  briefly  describe  the  major  components  of  this  approach  and  current 
efforts  to  apply  and  evaluate  its  utility  from  both  human-centered  and  cost-benefit  perspectives. 


Bradshaw,  J.  M.,  Beautement,  P.,  Breedy,  M.  R.,  Bunch,  L„  Drakunov,  S.  V.,  Feltovich,  P.  J.,  Hoffman,  R.  R., 
Jeffers,  R.,  Johnson,  M.,  Kulkarnt,  S.,  Lott,  J.,  Raj,  A.  K.,  Suri,  N.,  &  Uszok,  A.  (2004).  Chapter  12.  Making  Agents 
Acceptable  to  People.  In  N.  Zhong  and  J.  Liu  (Eds.),  Intelligent  Technologies  for  Information  Analysis:  Advances  in 
Agents,  Data  Mining,  and  Statistical  Learning.  Pp.  355-400. 
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-  Third  International  Central  and  Eastern  European  Conference  on  Multi-Agent  Systems  (Ceemas  2003).  Prague, 
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Summary  abstract: 

Because  ever  more  powerful  intelligent  agents  will  interact  with  people  in  increasingly  sophisticated  and  important 
ways,  greater  attention  must  be  given  to  the  technical  and  social  aspects  of  how  to  make  agents  acceptable  to  people. 
The  technical  challenge  is  to  devise  a  computational  structure  that  guarantees  that  from  the  technical  standpoint  all  is 
under  control.  The  social  challenge  is  to  ensure  that  agents  and  people  interact  gracefully  and  to  provide  reassurance 
to  people  that  all  is  working  according  to  plan.  In  this  chapter,  we  outline  our  efforts  to  address  some  of  these 
technical  and  social  concerns  through  the  use  of  a  policy-based  approach  as  implemented  in  the  KAoS  framework. 
From  a  technical  perspective,  we  want  to  be  able  to  help  ensure  the  protection  of  agent  state,  the  viability  of  agent 
communities,  and  the  reliability  of  the  resources  on  which  they  depend.  To  accomplish  this,  we  must  guarantee, 
insofar  as  is  possible,  that  the  autonomy  of  agents  can  always  be  bounded  by  explicit  enforceable  policy  that  can  be 
continually  adjusted  to  maximize  the  agents’  effectiveness  and  safety  in  both  human  and  computational 
environments.  From  a  social  perspective,  we  want  agents  to  be  designed  to  fit  well  with  how  people  actually  work 
together.  Explicit  policies  governing  human-agent  interaction,  based  on  careful  observation  of  work  practice  and  an 
understanding  of  current  social  science  research,  can  help  assure  that  effective  and  natural  coordination,  appropriate 
levels  and  modalities  of  feedback,  and  adequate  predictability  and  responsiveness  to  human  control  are  maintained. 
These  factors  are  key  to  providing  the  reassurance  and  trust  that  are  the  prerequisites  to  the  widespread  acceptance 
of  agent  technology  for  nontrivial  applications. 


Bradshaw,  J.  M.,  Cabri,  G.,  &  Montanari,  R.  (2003).  Taking  Back  Cyberspace.  IEEE  Computer,  89-92. 

Key  Words: 
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Advanced  Decision  Architectures 
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Journal  Article 

Summary  abstract: 

Certain  roles  in  society  carry  with  them  such  a  high  level  of  fiduciary  responsibility  that  the  appropriate 
organizations  carefully  monitor  and  control  them  through  licensing,  credentials,  and  periodic  formal  review.  To  the 
degree  that  our  personal  welfare  depends  on  such  people,  we  count  trustworthiness  the  most  important  prerequisite 
of  our  willingness  to  commit  aspects  of  our  lives  and  resources  into  their  hands.  For  similar  reasons,  people  expect  a 
high  level  of  trustworthiness  before  delegating  complex  and  important  tasks  to  software  agents. 


Bradshaw,  J.  M.,  Feltovich,  P.  J.,  Johnson,  M.,  Bunch,  L.,  Breedy,  M.,  Jung,  H.,  Lott,  J.,  &  Uszok,  A.  (2008). 
Coordination  in  Human- Agent-Robot  Teamwork.  In  In  Proceedings  of  2008  International  Symposium  on 
Collaborative  Technologies  and  Systems  (CTS  2008),  Special  Session  on  Collaborative  Robots  and  Human  Robot 
Interaction,  Irvine.  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Coordination  is  an  essential  ingredient  of  a  teamworkcentered  approach  to  autonomy.  In  this  paper,  we  discuss  some 
of  the  challenges  and  requirements  for  successful  coordination,  and  briefly  how  we  have  used  KAoS  HART  services 
to  support  coordination  in  a  multi-team  human  robot  field  exercise. 


Bradshaw,  J.  M.,  Jung,  H.,  Kulkarni,  S.,  Allen,  J.,  Bunch,  L.,  Chambers,  N.,  Feltovich,  P.,  Galescu,  L.,  Jeffers,  R., 
Johnson,  M.,  Taysom,  W.,  &  Uszok,  A.  (2004).  Toward  Trustworthy  Adjustable  Autonomy  and  Mixed-Initiative 
Interaction  in  KAoS.  In  In  Proceedings  of  AAMAS  2004  Trust  workshop  (pp.  9-20).  New  York  City,  NY. 
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Summary  abstract: 

Trust  is  arguably  the  most  crucial  aspect  of  agent  acceptability.  Many  aspects  of  trust  can  be  addressed  through 
policy.  Policies  are  a  means  to  dynamically  regulate  the  behavior  of  system  components  without  changing  code  or 
requiring  the  cooperation  of  the  components  being  governed.  By  changing  policies,  a  system  can  be  continuously 
adjusted  to  accommodate  variations  in  externally  imposed  constraints  and  environmental  conditions.  In  this  paper 
we  describe  some  important  dimensions  relating  to  autonomy  and  give  examples  of  how  these  dimensions  might  be 
adjusted  in  order  to  enhance  performance  of  human-agent  teams.  We  then  offer  a  definition  of  mixed-initiative 
interaction  and  give  examples  of  relevant  policies.  We  introduce  Kaa,  the  KAoS  adjustable  autonomy  component. 
Finally,  we  provide  a  brief  comparison  with  two  other  implementations  of  adjustable  autonomy  concepts. 


Bradshaw,  J.  M.,  Jung,  H.,  Kulkarni,  S.,  Johnson,  M.,  Feltovich,  P.,  Allen,  J.,  Bunch,  L.,  Chambers,  N.,  Galescu,  L., 
Jeffers,  R.,  Suri,  N.,  Taysom,  W.,  &  Uszok,  A.  (2005).  Kaa:  Policy-Based  Explorations  of  a  Richer  Model  for 
Adjustable  Autonomy.  In  In  Proceedings  of  Fourth  International  Joint  Conference  on  Autonomous  Agents  and 
Multi-Agent  Systems  (AAMAS  05)  Utrecht  University,  The  Netherlands. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Categories  and  Subject  Descriptors 

1.2.11  [Artificial  Intelligence]:  Distributed  Artificial  Intelligence 
-  intelligent  agents,  multiagent  systems. 

General  Terms 

Performance,  Reliability,  Human  Factors 
Keywords 

Adjustable  Autonomy,  Policy,  Trust,  Human-agent  Teamwork, 

KAoS,  Kaa,  OWL 

Summary  abstract: 

Though  adjustable  autonomy  is  hardly  a  new  topic  in  agent  systems,  there  has  been  a  general  lack  of  consensus  on 
terminology  and  basic  concepts.  In  this  paper,  we  describe  the  multi-dimensional  nature  of  adjustable  autonomy  and 
give  examples  of  how  various  dimensions  might  be  adjusted  in  order  to  enhance  performance  of  human-agent  teams. 
We  then  introduce  Kaa  (KAoS  adjustable  autonomy),  which  extends  our  previous  work  on  KAoS  policy  and  domain 
services  to  provide  a  policy-based  capability  for  adjustable  autonomy  based  on  this  richer  notion  of  adjustable 
autonomy.  The  current  implementation  of  Kaa  uses  a  combination  of  ontologies  represented  in  OWL  and  influence- 
diagram-based  decision-theoretic  algorithms  to  determine  what  if  any  changes  should  be  made  in  agent  autonomy  in 
a  given  context.  We  have  demonstrated  Kaa  as  part  of  ONR  sponsored  research  to  improve  naval  de-mining 
operations  through  more  effective  human-robot  interaction.  A  brief  comparison  among  alternate  approaches  to 
adjustable  autonomy  is  provided. 


Bradshaw,  J.  M.,  Jung,  H.,  Kulkarni,  S.,  Johnson,  M.,  Feltovich,  P.,  Allen,  J.,  Bunch,  L.,  Chambers,  N.,  Galescu,  L., 
Jeffers,  R.,  Suri,  N.,  Taysom,  W.,  &  Uszok,  A.  (2005).  Toward  Trustworthy  Adjustable  Autonomy  in  Kaos.  In  R. 
Falcone  (Ed.),  Trusting  Agents  for  Trustworthy  Electronic  Societies.  Berlin:  Springer-Verlag  Berlin  Heidelberg. 
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Summary  abstract: 

Trust  is  arguably  the  most  crucial  aspect  of  agent  acceptability.  At  its  simplest  level,  it  can  be  characterized  in  terms 
of  judgments  that  people  make  concerning  three  factors:  an  agent's  competence,  its  benevolence,  and  the  degree  to 
which  it  can  be  rapidly  and  reliably  brought  into  compliance  when  things  go  wrong.  Adjustable  autonomy  consists 
of  the  ability  to  dynamically  impose  and  modify  constraints  that  affect  the  range  of  actions  that  the  humanagent 
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team  can  successfully  perform,  consistently  allowing  the  highest  degrees  of  useful  autonomy  while  maintaining  an 
acceptable  level  of  trust.  Many  aspects  of  adjustable  autonomy  can  be  addressed  through  policy.  Policies  are  a 
means  to  dynamically  regulate  the  behavior  of  system  components  without  changing  code  or  requiring  the 
cooperation  of  the  components  being  governed.  By  changing  policies,  a  system  can  be  adjusted  to  accommodate 
variations  in  externally  imposed  constraints  and  environmental  conditions.  In  this  paper  we  describe  some  important 
dimensions  relating  to  autonomy  and  give  examples  of  how  these  dimensions  might  be  adjusted  in  order  to  enhance 
performance  of  human-agent  teams.  We  introduce  Kaa  (KAoS  adjustable  autonomy)  and  provide  a  brief  comparison 
with  two  other  implementations  of  adjustable  autonomy  concepts. 


Bradshaw,  J.  M.,  Sierhuis,  M.,  Acquisti,  A.,  Feltovich,  P.,  Hoffman,  R.,  Jeffers,  R.,  Suri,  N.,  Uszok,  A.,  &  Hoof,  R. 
V.  (2003).  Living  with  Agents  and  Liking  It:  Addressing  the  Technical  and  Social  Acceptability  of  Agent 
Technology.  In  In  Proceedings  of  Spring  2003  AAAI  Symposium  on  Human- Agent  Interaction  in  Complex 
Environments.  Menlo  Park,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  summarizes  our  efforts  to  address  some  of  the  technical  and  social  aspects  of  agent  design  for  increased 
human  acceptability.  From  a  technical  perspective,  we  want  to  be  able  to  ensure  the  protection  of  agent  state,  the 
viability  of  agent  communities,  and  the  reliability  of  the  resources  on  which  they  depend.  To  accomplish  this,  we 
must  guarantee  insofar  as  possible  that  the  autonomy  of  agents  can  always  be  bounded  by  explicit  enforceable  policy 
that  can  be  continually  adjusted  to  maximize  their  effectiveness  and  safety  in  both  human  and  computational 
environments.  From  a  social  perspective,  we  want  agents  to  be  designed  so  as  to  fit  well  with  how  people  actually 
work  together.  Explicit  policies  governing  human-agent  interaction  based  on  careful  observation  of  work  practice 
and  an  understanding  of  current  social  science  research  can  help  assure  that  effective  and  natural  coordination, 
appropriate  levels  and  modalities  of  feedback,  and  adequate  predictability  and  responsiveness  to  human  control  are 
maintained.  We  see  these  technical  and  social  factors  as  key  to  providing  the  reassurance  and  trust  that  are 
prerequisite  to  the  widespread  acceptance  of  agent  technology  for  nontrivial  applications. 


Bradshaw,  J.  M.,  Suri,  N.,  Breedy,  M.,  Canas,  A.,  Davis,  R.,  Ford,  K.,  Hoffman,  R.,  Jeffers,  R.,  Kulkarni,  S.,  Lott,  J., 
Reichherzer,  T.,  &  Uszok,  A.  (2002).  Terraforming  Cyberspace.  In  In  Proceedings  of  Workshop  on  Process 
Coordination  and  Ubiquitous  Computing.  Orlando:  University  of  Central  Florida. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

During  the  1940s,  under  the  pseudonym  of  Will  Stewart,  Jack  Williamson  published  a  series  of  fictional  stories 
describing  a  process  for  attaching  atmospheres  to  planets  in  order  to  make  them  capable  of  sustaining  life. 
‘Terraforming,’  the  term  he  coined  for  this  activity  was  first  picked  up  by  other  science  fiction  writers.  Eventually,  it 
captured  the  imagination  of  a  small  but  zealous  core  of  scientists,  space  advocacy  groups,  and  segments  of  the 
public  who  began  focusing  on  Mars  as  the  most  likely  target  for  transformation  and  eventual  colonization.  The  May 
1991  issue  of  Life  Magazine  ran  a  cover  story  describing  a  150-year  plan  for  a  Martian  metamorphosis  through 
orbiting  solar  reflectors  that  would  melt  polar  water,  surface  factories  that  would  produce  needed  gases  in  the 
atmosphere,  and  the  ultimate  planting  of  hearty  plant  species  as  the  temperature  approached  the  freezing  point  of 
water.  Today  many  articles,  books,  and  Web  sites  continue  to  develop  the  theme. 


Branlat,  M.,  Fern,  L.,  Voshell,  M.,  &  Trent,  S.  (2009).  Understanding  Coordination  Challenges  in  Urban 
Firefighting:  A  Study  of  Critical  Incident  Reports  In  In  Proceedings  of  53rd  Annual  Meeting  of  the  Human  Factors 
and  Ergonomics  Society,  San  Antonio,  TX. 
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ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  explores  the  central  role  played  by  coordination  during  fire  incident  response.  Following  preliminary 
investigations  based  on  other  methods,  the  authors  analyzed  a  corpus  of  29  critical  incident  reports  produced  by  one 
fire  department’s  safety  services.  Urban  firefighting  is  a  complex  domain  in  which  members’  activities  need  to  be 
highly  synchronized  in  order  to  reach  common  operational  and  safety  goals.  As  the  work  environment  challenges  the 
capacity  of  firefighters  to  coordinate  efficiently,  coordination  breakdowns  might  occur,  creating  safety  threats  as 
role  interdependencies  become  difficult  to  manage. 


Brewer,  I.,  McNeese,  M.  D.,  Frazier,  T.  G.,  Fuhrmann,  S.,  &  Terrell,  I.  S.  (2005).  Expanding  Team  Knowledge 
Elicitation  through  Procedural,  Temporal,  and  Strategic  Elements.  In  In  Proceedings  of  49th  Annual  Meeting  of  the 
Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Current  research  within  knowledge  elicitation  and  cognitive  task  analysis  (see  Schraagen  et  al.  2000  for  multiple 
examples)  is  typically  applied  to  individual  activities  rather  than  teamwork.  Yet,  when  analysts  conduct  cognitive 
fieldwork,  the  possibility  exists  to  also  work  with  groups  of  experts.  In  some  cases,  these  groups  may  be  intact  (as  a 
“workgroup”)  and  operate  as  a  cohesive  unit  in  processing  information  and  requests.  In  other  cases,  experts  may 
work  individually  but  be  brought  together  to  share  their  collective  knowledge  on  given  issues  and  aspects  of  the 
cognitive  task  analysis.  Because  teamwork  is  absolutely  important  in  many  facets  of  complex  problems  (especially 
in  encountering  emergency  situations  that  require  joint  coordination,  communication,  and  control  of  resources),  it  is 
useful  to  engage  in  team  knowledge  elicitation  for  either  case  mentioned. 


Bunch,  L.,  Bradshaw,  J.  M.  &  Young,  C.  O.  (2008).  Policy -Governed  Information  Exchange  in  a  Us  Army 
Operational  Scenario.  In  In  Proceedings  of  2008  IEEE  Conference  on  Policy,  Palisades,  NY. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  authors  are  investigating  how  emerging  policy  and  semantic  web  technologies  can  be  used  to  help  provide  the 
best  set  of  available  tactical  information  to  the  Soldier  in  the  field.  In  this  initial  effort.  Researchers  from  the  U.S. 
Army  Research  Labs  (ARL)  and  the  Florida  Institute  for  Human  and  Machine  Cognition  (IHMC)  have  developed  a 
system  that  demonstrates  the  potential  of  these  technologies  in  a  small-scale  U.S.  Army  mockup  scenario.  The 
system  represents  and  reasons  about  domain-specific  policies  to  help  recognize  what  documents  the  end  Soldier  is 
allowed  to  receive  given  the  current  mission  context.  The  system  also  relies  on  policies  to  help  recognize  when 
appropriate  human  approval  can  be  obtained  or  a  specific  transformation  of  the  information  can  be  performed  to 
allow  the  information  to  be  sent.  Semantic  web  technologies  are  further  used  to  describe  the  properties  and  features 
of  each  document  and  relate  these  features  to  mission  contexts  in  which  the  information  is  likely  to  be  appropriate. 
The  result  is  a  compelling  demonstration  of  the  role  that  policies  and  semantic  web  technologies  can  play  in 
promoting  the  Army’s  need  to  share  information  while  remaining  vigilant  of  the  requirements  to  protect  methods 
and  sources. 


Bunch,  L.,  Bradshaw,  J.  M.,  Johnson,  M.,  Lott,  J.,  J.Feltovich,  P.,  Suri,  N.,  Carvalho,  M.,  Tokarcik,  L.,  Winkler,  R., 
&  Metu,  S.  (2009).  Chapter  4.  Regulating  the  Exchange  of  Tactical  Military  Information  Using  the  Kaos  Policy 
Services  Framework.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter: 
Foundations  and  Technology.  Section  1.  Collecting,  Processing  and  Distributing  Battlefield  Information: 
48hrBooks  by  Alion  Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the 
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Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research 
Management  Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1100 
New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr- 
webtest. alionscience.com/AdaCta/ Agenda,  aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

Sharing  tactical  information  among  joint  forces,  coalition  partners,  and  non-government  organizations  (NGOs)  can 
be  critical  to  successful  and  safe  operations.  Unfortunately,  the  need  to  share  information  is  sometimes  at  odds  with 
the  sensitivity  of  battlefield  information.  Frequently,  the  flow  of  data  across  heterogeneous  channels  and  among 
diverse  groups  must  be  tightly  regulated.  This  presents  significant  operational  challenges,  including  successful 
negotiation  of  the  multi-level,  interdependent,  and  sometimes  conflicting  agency  and  interagency  aims,  governed 
formally  by  policy  and  chains  of  command,  and  informally  by  cultural  and  organizational  norms. 

IFlMCand  ARL  are  collaborating  to  develop  software  systems  in  which  information  is  passed  seamlessly  across 
multiple  organizations,  networks,  and  individuals  while  respecting  complex  and  possibly  conflicting  sets  of 
policies.  In  support  ofthis  objective,  we  have  created  capabilities  that  allow  us  to  automatically  and  dynamically 
identify,  in  a  contextsensitive  way,  when  requisite  operational  and  situational  imperatives  warrant  the  automated  (or 
semi-automated)  declassification  and  dissemination  of  sensitive  but  perishable  combat  data.  Moreover,  we  are 
exploring  mechanisms  for  abstracting  and  transforming  information  to  enable  it  to  fulfill  requirements  for  sharing, 
for  example,  by  redacting  the  content  or  metadata  of  messages  or  by  delaying  their  delivery  until  a  critical  situation 
is  over.  By  providing  automated  mechanisms  to  help  identify  relevant  sensitive  but  perishable  data  in  the  Collateral 
Space  that  can  be  released,  transformed,  or  delayed  with  appropriate  levels  of  human  oversight  and  approval,  lives 
can  be  saved  and  time  wasted  in  performing  these  tedious  tasks  manually  can  be  significantly  reduced  or  eliminated. 
Our  approach  includes  the  use  of  an  extensible  standards-based  policy  representation  to  specify  formal  policy 
statements  about  the  tactical  domain.  The  KAoS  Policy  and  Domain  Services  framework  provides  a  graphical 
interface  for  quickly  constructing  policies  in  readable  English  in  a  point-and-click  fashion  at  run-time.  KAoS  also 
provides  innovative  formal  ontology-based  policy  analysis  and  deconfhction  mechanisms  and  highly-efficient 
mechanisms  for  policy  monitoring  and  enforcement.  It  also  includes  many  powerful  features  that  enhance  its 
usefulness  in  real  battlefield  situations  (e.g.,  support  for  standalone  or  disconnected  operations,  spatial  and  temporal 
reasoning).  We  integrate  this  policy  reasoning  capability  with  distributed  middleware  components  capable  of 
enforcing  such  policies  in  a  MANET  (mobile  adhoc  network)  environment 


Bunch,  L.,  Breedy,  M.,  Bradshaw,  J.  M.,  Carvalho,  M.,  Danks,  D.,  &  Suri,  N.  (2005).  Flexible  Automated 
Monitoring  and  Notification  for  Complex  Processes.  In  In  Proceedings  of  IEEE  International  Conference  on 
Networking,  Sensing,  and  Control,  Tucson,  AZ. 
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Summary  abstract: 

Automated  monitoring  for  complex  systems  such  as  the  space  shuttle  fueling  and  launch  process  can  increase  the 
effectiveness  of  operators  in  detecting  abnormal  conditions  before  become  hazardous  to  the  point  they  may 
compromise  the  mission.  We  have  developed  the  KAoS  Reactive  Monitoring  and  Event  Notification  (KARMEN) 
multi-agent  system  to  allow  users  to  describe  and  change  monitoring  conditions  at  any  time  in  order  to  effectively 
monitor  such  processes  and  appropriately  notify  key  operations  personnel  about  off-nominal  conditions.  These 
notification  actions  are  overseen  by  policies  that  contain  process  requirements  and  users  preferences. 


Bunch,  L.,  Breedy,  M.,  Bradshaw,  J.  M.,  Carvalho,  M.,  &  Suri,  N.  (2005).  Karmen:  Multi-Agent  Monitoring  and 
Notification  for  Complex  Processes.  In  In  Proceedings  of  HoloMAS  Conference  2005,  Copenhagen,  Denmark. 
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ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Early  and  consistent  detection  of  abnormal  conditions  is  important  to  the  safe  and  efficient  operation  of  complex 
industrial  processes.  Our  research  focuses  on  enabling  the  operators  and  engineers  who  control  and  maintain  such 
systems  tO, describe  process  conditions  to  software  agents,  deploy  such  agents  to  continuously  monitor  live  process 
data,  and  receive  appropriate  notification  from  their  personal  agents  concerning  the  process  state.  The  resulting 
dynamic  population  of  monitoring  agents  is  managed  by  our  agile  computing  framework  according  to  policies  that 
define  computing  and  networking  resource  restrictions  as  well  as  user  notification  requirements  and  preferences. 


Burke,  C.  S.,  Fiore,  S.  M.,  &  Salas,  E.  (2003).  Chapter  5.  The  Role  of  Shared  Cognition  in  Enabling  Shared 
Leadership  and  Team  Adaptability  In  C.L.  Pearce  and  J.A.  Conger  (Eds.),  Shared  Leadership:  Reframing  the  How's 
and  Why's  of  Leadership.  Pp.  103-122.  Thousand  Oaks,  CA:  Sage  Publishers. 

Key  Words: 
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Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Organizations  are  increasingly  using  teams  as  a  predominant  strategy  when  faced  with  environments  characterized 
by  complexity  and  ambiguity.  These  teams  are  typically  composed  of  two  or  more  members  who  work 
interdependently  and  adaptively  to  accomplish  a  common  goal  or  mission.  Such  teams  typically  have  specific  role 
assignments  and  they  often  disband  once  the  goal  is  reached  or  mission  is  completed  (Salas,  Dickinson,  Converse,  & 
Tannenbaum,  1992).  Moreover,  due  to  the  complicated  nature  of  the  tasks  facing  those  in  industry  and  the  military, 
these  teams  are  increasingly  becoming  cross-functional  (see  Northcraft,  Polzer,  Neale,  &  Kramer,  1995;  Knight  et 
al.,  1999).  Despite  the  proposed  benefits  of  work  teams  (see  Swezey  &  Salas,  1992),  many  real  world  examples  have 
illustrated  that  they  are  not  always  effective  (Hackman,  1990).  Key  among  the  reasons  for  the  failure  of  teams  to  live 
up  to  their  potential  are  (a)  the  failure  to  smoothly  coordinate  member  action  and  (b)  a  lack  of  effective  leadership  to 
guide  the  coordination  process.  In  fact,  some  have  argued  that  the  team  leader's  failure  to  guide  and  structure  team 
experiences  that  facilitate  the  development  and  maintenance  of  coordinative,  adaptive  action  can  be  a  key  factor  in 
ineffective  performance  (Stewart  &  Manz,  1995).  Clearly,  coordinative,  adaptive  action  is  vital  to  successful  team 
performance. 


Burke,  C.  S.,  Fowlkes,  J.  E.,  Wilson,  K.  A.,  &  Salas,  E.  (2003).  A  Concept  of  Soldier  Adaptability:  Implications  for 
Design  and  Training.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and 
Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University 
College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 
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Summary  abstract: 

Adaptability  has  been  argued  to  be  a  defining  characteristic  of  effective  teams  (Cannon-Bowers,  Tannenbaum, 

Salas,  &  Volpe,  1995).  Additionally,  as  military  missions  are  increasingly  varied,  complex,  and  dynamic  requiring 
soldiers  to  complete  their  missions  in  the  presence  of  asymmetric  threats  adaptability  is  ever  more  important. 
Despite  the  need  for  adaptive  action  and  adaptive  teams,  recent  evidence  from  the  operational  literature  has 
suggested  several  barriers  to  adaptive  team  performance  in  the  military.  Many  of  these  barriers  can  be  eliminated  or 
mitigated  through  training  or  system  design.  However,  there  has  yet  to  be  much  attention  given  or  guidance  offered 
for  training  soldier  adaptability  or  designing  systems  to  facilitate  soldier  adaptability.  Thereby,  the  purpose  of  the 
present  paper  is  to  illustrate  a  theoretical  concept  of  adaptability,  the  key  tenets  that  form  this  concept,  and  a 
translation  of  these  tenets  into  training  and  design  implications.  We  also  offer  a  set  of  conceptually  derived  training 
and  design  guidelines,  a  subset  of  which  are  being  empirically  validated. 
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Burke,  C.  S.,  Fowlkes,  J.  E.,  Wilson,  K.  A.,  &  Salas,  E.  (2003).  A  Concept  of  Soldier  Adaptability:  Implications  for 
Design  and  Training.  In  In  Proceedings  of  Annual  Conference  for  the  Collaborative  Technology  Alliance  Program, 
pp.  143-148.  Baltimore,  MD. 


Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Adaptability  has  been  argued  to  be  a  defining  characteristic  of  effective  teams  (Cannon-Bowers,  Tannenbaum, 
Salas,  &  Volpe,  1995).  Additionally,  as  military  missions  are  increasingly  varied,  complex,  and  dynamic  requiring 
soldiers  to  complete  their  missions  in  the  presence  of  asymmetric  threats  adaptability  is  ever  more  important. 
Despite  the  need  for  adaptive  action  and  adaptive  teams,  recent  evidence  from  the  operational  literature  has 
suggested  several  barriers  to  adaptive  team  performance  in  the  military.  Many  of  these  barriers  can  be  eliminated  or 
mitigated  through  training  or  system  design.  However,  there  has  yet  to  be  much  attention  given  or  guidance  offered 
for  training  soldier  adaptability  or  designing  systems  to  facilitate  soldier  adaptability.  Thereby,  the  purpose  of  the 
present  paper  is  to  illustrate  a  theoretical  concept  of  adaptability,  the  key  tenets  that  form  this  concept,  and  a 
translation  of  these  tenets  into  training  and  design  implications.  We  also  offer  a  set  of  conceptually  derived  training 
and  design  guidelines,  a  subset  of  which  are  being  empirically  validated. 


Burke,  C.  S.,  Stagl,  K.  C.,  Salas,  E.,  Pierce,  L.,  &  Kendall,  D.  (2007).  Understanding  Team  Adaptation:  A 
Conceptual  Analysis  and  Model.  Journal  of  Applied  Psychology,  91(6),  1189-1207. 
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teams,  teamwork,  team  adaptation,  adaptability,  team  effectiveness 
Summary  abstract: 

This  endeavor  provides  a  multidisciplinary,  multilevel,  and  multiphasic  conceptualization  of  team  adaptation  with 
theoretical  roots  in  the  cognitive,  human  factors,  and  industrial-  organizational  psychology  literature.  Team 
adaptation  and  the  emergent  nature  of  adaptive  team  performance  are  defined  from  a  multilevel,  theoretical 
standpoint.  An  input-throughput-  output  model  is  advanced  to  illustrate  a  series  of  phases  unfolding  over  time  that 
constitute  the  core  processes  and  emergent  states  underlying  adaptive  team  performance  and  contributing  to  team 
adaptation.  The  cross-level  mixed-determinants  model  highlights  team  adaptation  in  a  nomological  network  of 
lawful  relations.  Testable  propositions,  practical  implications,  and  directions  for  further  research  in  this  area  are  also 
advanced. 


Carley,  K.  M.  (2007).  Dynamic  Network  Analysis  for  Counter-Terrorism.  In  In  Proceedings  of  Kilby  Lecture  at  the 
GOMATech  07,  Lake  Buena  Vista,  FL. 

Key  Words: 
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Summary  abstract: 

Dynamic  network  analysis  (DNA)  is  an  emergent  field  centered  on  the  collection,  analysis,  understanding  and 
prediction  of  dynamic  relations  (such  as  who  talks  to  whom  and  who  knows  what)  and  the  impact  of  such  dynamics 
on  individual  and  group  behavior.  DNA  facilitates  reasoning  about  real  groups  as  complex  dynamic  systems  that 
evolve  over  time.  In  this  chapter,  the  basic  tenets  of  DNA  are  described  and  contrasted  with  those  of  Social  Network 
Analysis  and  Link  Analysis.  Some  of  the  basic  techniques  are  then  illustrated  through  the  analysis  of  data  on  al 
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Qaeda.  Technology  described  enables  the  analyst  to  identify  vulnerabilities  in  the  terrorist  network  and  to  assess 
how  that  network  might  change  in  response  to  strategic  interventions. 


Carley,  K.  M.  (2007).  A  Dynamic  Network  Approach  to  the  Assessment  of  Terrorist  Groups  and  the  Impact  of 
Alternative  Courses  of  Action.  In  In  Proceedings  of  RTO,  NATO.  DTIC  ADM002067. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Dynamic  network  analysis  (DNA)  is  an  emergent  field  centered  on  the  collection,  analysis,  understanding  and 
prediction  of  dynamic  relations  among  various  entities  such  as  actors,  events  and  resources  and  the  impact  of  such 
dynamics  on  individual  and  group  behaviour.  DNA  facilitates  reasoning  about  terrorist  groups  as  complex  dynamic 
networked  systems  that  evolve  over  time.  An  interoperable  DNA  toolchain  for  collecting  data  on,  assessing  the 
network  of,  and  forecasting  changes  in  that  network  is  presented.  The  use  of  these  DNA  tools  to  asses  a  terrorist 
group  is  then  demonstrated  using  open  source  data.  Key  techniques  are  demonstrated  using  a  dataset  collected  on 
terror  networks.  Techniques  demonstrated  include  those  for  identification  of  an  actor’s  sphere  of  influence, 
emergent  leaders,  and  paths  among  critical  actors,  and  metrics  for  assessing  the  potential  immediate  and  near  term 
impact  of  various  courses  of  action. 


Carley,  K.  M.  (2009).  Dynamic  Network  Analysis.  In  In  Proceedings  of  NRC  workshop  on  Social  Network 
Modeling  and  Analysis,  Ron  Breiger  and  Kathleen  M.  Carley  (Eds.),  National  Research  Council. 

Key  Words: 
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Summary  abstract: 

Dynamic  network  analysis  (DNA)  varies  from  traditional  social  network  analysis  in  that  it  can  handle  large  dynamic 
multi-mode,  multi-link  networks  with  varying  levels  of  uncertainty.  DNA,  like  quantum  mechanics,  would  be  a 
theory  in  which  relations  are  probabilistic,  the  measurement  of  a  node  changes  its  properties,  movement  in  one  part 
of  the  system  propagates  through  the  system,  and  so  on.  However,  unlike  quantum  mechanics,  the  nodes  in  the 
DNA,  the  atoms,  can  learn.  An  approach  to  DNA  is  described  that  builds  DNA  theory  through  the  combined  use  of 
multi-agent  modeling,  machine  learning,  and  meta-matrix  approach  to  network  representation.  A  set  of  candidate 
metric  for  describing  the  DNA  are  defined.  Then,  a  model  built  using  this  approach  is  presented.  Results  concerning 
the  evolution  and  destabilization  of  networks  are  described. 


Carley,  K.  M.,  Martin,  M.  K.,  &  Hancock,  J.  P.  (2009).  Chapter  1.  Dynamic  Network  Analysis  Applied  to 
Experiments  from  the  Decision  Architectures  Research  Environment.  In  P.  McDermott  and  L.  Allender  (Eds.), 
Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  I.  Collecting, 
Processing  and  Distributing  Battlefield  Information:  48hrBooks  by  Alion  Science  &  Technology  Corporation  under 
contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and 
Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative 
Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the 
Soldier  Decision  Maker.  July  22-23,  2009.  1100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003. 
http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 
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Summary  abstract: 

ADA  CTA  research  is  producing  experimental  results  from  simulations  of  intermeshed  networks  of  warfighters  and 
battlefield  surveillance'  assets.  These  networks  form  a  complex  system  with  behaviors  that  emerge  from  patterns  of 
interaction  among  constituent  entities.  The  simulated  interactions  are  spatially  situated,  temporally  distributed 
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communications  among  people,  robots,  and  software  agents.  In  general  terms,  the  complex  system  -  parts  of  which 
we  address  in  this  paper  -  can  be  conceptualized  as  a  two-level  meta-network  that  includes  interactions  among 
human  agents  at  one  level,  interactions  among  artificial  agents  at  another  level,  and  cross-level  interactions  between 
human  and  artificial  agents. 

The  issue  addressed  in  this  paper  is  Dynamic  Network  Analysis  (DNA)  of  system  behavior.  For  the  purposes  of  this 
chapter,  we  are  not  interested  in  examining  the  performance  of  one  system  relative  to  another.  Although  this  type  of 
comparative  analysis  can  be  useful  for  researchers  or  system  designers,  it  is  of  questionable  use  to  warfighters. 
Instead,  we  are  interested  in  analyses  that  produce  tactically  relevant,  actionable  results  that  highlight  the  strengths 
and  weaknesses  in  the  system  being  observed.  To  be  useful,  the  analytical  results  must  foster  tactical  insight  and 
stimulate  battlefield  decisions  that  prudently  influence  future  system  behavior. 

To  this  end,  we  describe  two  case  studies  that  apply  DNA  to  the  simulated  battlefield  data  being  generated  by 
experiments  in  the  Decision  Architectures  Research  Environment  (DARE).  The  first  case  involves  intercepts  of 
simulated  communications  among  human  agents,  which  we  frame  as  an  exercise  in  adversarial  reasoning.  The 
second  case  involves  simulated  communications  among  surveillance  assets  (i.e.,  software  agents  and  robots),  which 
we  frame  as  an  exercise  in  understanding  the  automated  control  of  a  Persistent  Coordinated  Video  Surveillance 
(PCVS)  system.  Together,  we  believe  the  case  studies  demonstrate  how  DNA  (Carley,  2002)  can  foster  tactical 
insight  in  complex  multi-entity  scenarios.  They  also  demonstrate  that  the  combination  of  DNA  techniques  required 
for  tactical  insight  may  vary  according  to  the  type  of  network  being  analyzed.  Finally,  they  show  how  DNA  assists 
in  the  development,  understanding  and  tuning  of  software  agent  systems. 


Carroll,  M.,  Josephson,  J.  R.  &  Russell,  J.  L.  (2007).  Tradeoffs  on  the  Efficient  Frontier  for  the  Disruption  of 
Networks.  In  In  Proceedings  of  IEEE  Symposium  on  Computational  Intelligence  in  Multi-Criteria  Decision-Making 
(MCDM  2007),  (http://www.ieeecis-multidecision.org/)  Honolulu,  Hawaii. 
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Summary  abstract: 

A  communications  network  is  represented  as  a  graph  of  flow  capacities.  We  study  the  problem  of  finding  good 
network  disruption  attacks  or  target  sets,  i.e.,  a  subset  of  vertices  or  edges  that,  once  removed,  impede 
communication  between  particular  nodes.  Multiple  costs  are  associated  with  removing  vertices  or  edges.  Success  in 
disrupting  communications  is  traded  off  against  the  costs  of  the  attack  plans:  the  efficient  frontier  of  attacks  is 
estimated,  and  the  results  are  studied  in  crosslinked  diagrams.  A  multicriterial  genetic  algorithm  is  used  to  discover 
good  plans  for  disrupting  the  communications  network,  where  the  genes  correspond  to  nodes  or  links  to  be  attacked. 
The  genetic  algorithm  is  seeded  with  an  initial  population  of  single-target  genomes,  one  for  each  potential  target. 
Multi-target  attacks  may  be  generated  by  breeding.  Being  on  the  efficient  frontier  guarantees  a  genome’s  survival  to 
the  next  generation,  so  the  population  size  is  allowed  to  vary.  The  results  are  studied  in  interactive  diagrams  and  in 
an  “aggregate  view”  of  the  resulting  population.  Good  attacks  were  found  relatively  rapidly,  and  the  aggregate  view 
revealed  significant  targets. 


Carvalho,  M.,  Suri,  N.,  Shurbanov,  V.  and  Lloyd,  E.  (2007).  A  Cross-Layer  Network  Substrate  for  the  Battlefield. 
In  In  Proceedings  of  25th  Army  Science  Conference,  Orlando,  FL. 

Key  Words: 
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Summary  abstract: 

This  paper  describes  the  initial  design  and  implementation  of  a  cross-layer  communications  substrate  for  tactical 
networks.  Traditional  crosslayer  strategies  for  MANETs  often  rely  on  the  direct  interaction  between  neighbor  layers 
in  the  communications  stack.  We  propose  a  different  approach,  where  all  lower  layers  (PHY,  MAC  and  NET) 
directly  interact  with  the  overlying  applications  (or  communications  middleware).  In  this  work,  we  discuss  some  of 
the  requirements  for  cross-layer  support  in  a  tactical  environment.  We  also  introduce  our  proposed  design  for  a 
cross-layer  communications  substrate  for  such  environments,  concluding  the  paper  with  a  brief  description  of  our 
current  proof-of-concept  implementation  and  future  research  proposal. 
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Carvalho,  M.,  Granados,  A.,  Brothers,  A.,  Hanna,  J.  P.,  &  Turck,  K.  (2009).  A  Cross-Layer  Communications 
Substrate  for  Tactical  Information  Management  Systems.  In  In  Proceedings  of  MILCOM,  San  Diego. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  this  paper  we  introduce  XLayer,  a  cross-layer  communications  substrate  for  tactical  Information  Management 
Systems  which  enable  nodes  on  a  radio  network  to  seamlessly  communicate  with  nodes  on  different  heterogeneous 
networks.  While  conventional  cross-layer  strategies  for  tactical  environments  tend  to  focus  on  the  localized 
optimization  between  neighbor  layers  ofthe  communications  stack,  our  approach  focuses  on  the  interface  between 
middleware  and  the  underlying  communications  infrastructure. 

The  XLayer  communications  substrate  leverages  native  information  and  services  available  at  the  tactical 
communications  infrastructure  to  improve  the  functionally  and  capabilities  of  overlay  applications  and  middleware. 
The  XLayer  also  provides  the  necessary  interfaces  and  mechanisms  to  enable  application-driven  requirements  to 
parameterize  and  regulate  the  operation  ofthe  underlying  communications  infrastructure. 

After  a  brief  description  of  the  target  environment  and  system  requirements  we  will  introduce  the  proposed  design 
for  the  cross-layer  communications  substrate,  highlighting  specialized  controllers  and  adaptors  for  communication 
interfaces  and  tactical  radios.  We  will  then  introduce  new  cross-layer  strategies  for  discovery,  routing  and  transport 
targeted  to  Information  Management  System  (IMS)-support,  followed  by  our  NS-2  simulation  results,  analysis,  and 
conclusions. 


Carvalho,  M.,  Granados,  A.,  Perez,  C„  Arguedas,  M.,  Winkler,  R.,  Kovach,  J.,  &  Choy,  S.  (2009).  Chapter  5. 

Xlayer:  A  Cross-Layer  Communications  Substrate  for  Tactical  Environments.  In  P.  McDermott  and  L.  Allender 
(Eds.),  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  I.  Collecting, 
Processing  and  Distributing  Battlefield  Information:  48hrBooks  by  Alion  Science  &  Technology  Corporation  under 
contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and 
Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative 
Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the 
Soldier  Decision  Maker.  July  22-23,  2009.  1100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003. 
http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

Tactical  and  mobile  ad  hoc  networks  (MANETs)  are  key  technology  enablers  for  future  mission-critical 
communication  infrastructures  such  as  those  envisioned  for  disaster  relief  operations  and  military  missions. 

The  self-organizing  nature  of  MANETs  enables  the  on-demand  formation  of  temporary  communications  capabilities 
to  support  the  operation  of  mission-critical  applications  in  the  edges  of  main  communications  infrastructure,  a 
critical  capability  for  military  operations. 

However,  this  unprecedented  capability  comes  at  a  price.  The  stability  and  reliability  generally  observed  in  fixed 
communications  infrastructures,  which  are  often  leveraged  and  assumed  by  tactical  applications  and  decision 
architectures,  are  no  necessarily  available  in  MANETs  (at  least  at  similar  levels).  The  flexibility  that  enables  the  on- 
demand  formation  of  MANETs  also  creates  a  new  type  of  communications  environment,  one  that  is  dynamic  and 
significantly  more  resource  constrained. 

While  there  has  been  a  significant  amount  of  research  focused  primarily  on  building  new  MANET  technologies  and 
protocols  that  would  provide  better  predictability  and  reliability,  it  is  widely  accepted  today  that  a  practical  solution 
will  likely  require  a  better  interaction  between  application  and  communications  infrastructure.  The  traditional 
layered  approach  that  has  shielded  applications  from  the  underlying  networks  is  no  longer  applicable,  and  new 
challenge  has  emerged-how  to  provide  such  interface  (or  mutual  awareness)  while  avoiding  customized  stovepiped 
solutions. 
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The  notion  of  a  cross-layer  communications  substrate  for  tactical  battlefield  environments  described  in  this  work 
proposes  to  address  the  problem  by  enabling  a  two-way  interface  between  higher-level  applications,  middleware  or 
decision  architectures,  and  the  underlying  communications  infrastructure. 

In  this  work  we  summarize  our  research  for  the  development  of  XLayer,  a  cross-layer  communications  substrate 
designed  to  provide  and  support  communication  infrastructures  for  the  battlefield. 

The  goal  is  to  provide  a  modular  architecture  and  a  set  of  interfaces  that  1)  allows  applications  and  decision 
architecture  systems  to  better  adapt  to  and  leverage  the  characteristics  of  the  dynamic  communications  environment 
and,  2)  enables  the  underlying  communications  infrastructure  to  better  support  application  requirements  and 
constraints. 

In  support  of  the  application  and  decision  architecture  systems,  XLayer  monitors,  abstracts,  and  represents  the 
characteristics  and  capabilities  of  the  underlying  communications  infrastructure  so  applications  can  better  adapt  to 
changes  in  the  underlying  communications  environment  (by  re-allocating  resources,  for  example). 

In  support  of  the  communications  infrastructure,  applications  can  provide  information  about  resource  requirements 
(both  computational  and  communications)  or  utilization  patterns.  This  information  can  then  be  used  by  the 
underlying  communications  infrastructure  to  better  allocate  resources  and  capabilities  in  response  to  changes  that 
may  occur  either  on  demand  or  proactively,  based  on  explicit  application  requirement  patterns. 


Carvalho,  M.,  Perez,  C.,  &  Granados,  A.  (2009).  Dynamic  Gateway  Selection  for  Cross-Domain  Routing  with  the 
Xlayer  Communications  Substrate.  In  In  Proceedings  of  International  Workshop  on  Scalable  Ad  Hoc  and  Sensor 
Networks  (SASN  2009),  Saint  Petersburg,  Russia. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  describes  a  dynamic  gateway  selection  algorithm  for  cross-domain  routing.  The  proposed  cross-layer 
algorithm  is  designed  on  top  of  the  XLayer  communications  substrate.  In  the  context  of  this  work,  cross-domain 
routing  refers  to  seamlessly  routing  across  different  networks  running  different  routing  algorithms  without  pre¬ 
defined  gateways  or  a  common  underlaying  protocol.  We  first  provide  a  brief  introduction  of  the  XLayer 
communications  substrate  and  the  adaptive  routing  controller.  After  describing  the  algorithm  and  a  summary  of  the 
proposed  architecture,  a  proof-of-concept  implementation  is  evaluated  for  a  simple  scenario  developed  in  NS-2.  Our 
preliminary  results  are  encouraging  and  show  the  potential  of  our  proposed  strategy  for  dynamic  gateway  selection 
and  crossdomain  routing. 


Carvalho,  M.,  Suri,  N.,  &  Arguedas,  M.  (2006).  Mobile  Agent-Based  Communications  Middleware  for  Data 
Streaming  in  the  Battlefield.  In  In  Proceedings  of  MILCOM  2005  Conference,  Atlantic  City,  NJ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  this  paper  we  introduce  the  FlexFeedframework  in  the  context  of  military  combat  operations.  FlexFeed  realizes 
the  notion  ofAgile  Computing  for  data-streaming  and  implements  a  flexible,  robust  and  efficient  publishlsubscribe 
infrastructure  for  dynamic  ad  hoc  environments  under  resource  and  policy  constraints.  The  framework  uses  mobile 
software  agents  for  underlying  configuration  and  policy  enforcement.  The  paper  illustrates  the  effectiveness  of  the 
framework  with  quantitative  experiments  over  simulated  scenarios. 


Carvalho,  M.,  Suri,  N.,  Arguedas,  M.,  Rebeschini,  M.,  &  Breedy,  M.  (2007).  A  Cross-Layer  Communications 
Framework  for  Tactical  Environments.  In  In  Proceedings  of  MILCOM  2006,  Washington,  D.C.  Also  Invited  Talk 
at  BAE  Systems  and  Invited  talk  at  the  Federal  University  of  Brasilia,  Brazil. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
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Summary  abstract: 

In  this  paper  we  propose  a  cross-layer  approach  for  information  and  resource  sharing  between  the  routing,  transport 
and  application-level  components  in  tactical  networks.  Our  approach  is  different  than  traditional  crosslayer 
strategies  for  MANETs  in  that  itfocuses  on  the  interaction  of  each  layer  (MAC  and  NET)  directly  with  the  overlying 
applications  (or  middleware),  as  opposed  to  the  traditional  interactions  between  neighbor  layers  in  the  OSI  model.  In 
this  work,  'applications'  are  abstracted  by  a  data-aware  communication  framework  that  resides  on  top  ofthe  transport 
layer  and  interacts  will  all  sub-layers.  The  paper  describes  the  requirements  for  cross-layer  integration,  and 
introduces  our  proposed  approach.  A  proof-ofconcept  implementation  is  presentedfor  the  Agile  Computing 
Middleware  and  the  OLSR  routing  protocol. 


Cekova,  C.,  Chandrasekaran,  B.,  Josephson,  J.,  &  Pantaleev,  A.  (2006).  Simulation-Based  Planning  for 
Peacekeeping  Operations:  Selection  of  Robust  Plans.  In  The  International  Society  for  Optical  Engineering  (SPIE) 
Defense  and  Security  Symposium,  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
modeling,  simulation,  robustness,  selection  of  robust  plans,  course  of  action,  COA,  Course  of  Action  analysis 
Approach 

Summary  abstract: 

This  research  is  part  of  a  proposed  shift  in  emphasis  in  decision  support  from  optimality  to  robustness.  Computer 
simulation  is  emerging  as  a  useful  tool  in  planning  courses  of  action  (COAs).  Simulations  require  domain  models, 
but  there  is  an  inevitable  gap  between  models  and  reality  -  some  aspects  of  reality  are  not  represented  at  all,  and 
what  is  represented  may  contain  errors.  As  models  are  aggregated  from  multiple  sources,  the  decision  maker  is 
further  insulated  from  even  an  awareness  of  model  weaknesses.  To  realize  the  full  power  of  computer  simulations  to 
support  decision  making,  decision  support  systems  should  support  the  planner  in  exploring  the  robustness  oieGAs  in 
the  face  of  potential  weaknesses  in  simulation  models. 

This  paper  demonstrates  a  method  of  exploring  the  robustness  of  a  COA  with  respect  to  specific  model  assumptions 
about  whose  accuracy  the  decision  maker  might  have  concerns.  The  domain  is  that  of  peacekeeping  in  a  country 
where  three  different  demographic  groups  co-exist  in  tension.  An  external  peacekeeping  force  strives  to  achieve 
stability,  an  improved  economy,  and  a  higher  degree  of  democracy  in  the  country.  A  proposed  COA  for  such  a  force 
is  simulated  multiple  times  while  varying  the  assumptions.  A  visual  data  analysis  tool  is  used  to  explore  COA 
robustness.  The  aim  is  to  help  the  decision  maker  choose  a  COA  that  is  likely  to  be  successful  even  in  the  face  of 
potential  errors  in  the  assumptions  in  the  models. 


Cevher,  V.,  Guo,  F.,  Sankaranarayanan,  A.  C.,  &  Chellappa,  R.  (2007).  Joint  Acoustic-Video  Fingerprinting  of 
Vehicles,  Part  II.  In  In  Proceedings  of  ICASSP  2007. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Object  recognition,  pattern  recognition,  acoustic  applications,  acoustic  signal  processing,  intelligent  sensors 

Summary  abstract: 

In  this  second  paper,  we  first  show  how  to  estimate  the  wheelbase  length  of  a  vehicle  using  line  metrology  in  video. 
We  then  address  the  vehicle  fingerprinting  problem  using  vehicle  silhouettes  and  color  invariants.  We  combine  the 
acoustic  metrology  and  classification  results  discussed  in  Part  I  with  the  video  results  to  improve  estimation 
performance  and  robustness.  The  acoustic  video  fusion  is  achieved  in  a  Bayesian  framework  by  assuming 
conditional  independence  of  the  observations  of  each  modality.  For  the  metrology  density  functions,  Laplacian 
approximations  are  used  for  computational  efficiency.  Experimental  results  are  given  using  field  data. 


Chadwick,  R.,  Pazuchanics,  S.  L.,  &  Gillan,  D.  (2008).  Visual-Cognition  Challenges  for  the  Operation  of  Unmanned 
Ground  Vehicles  SPIE  Newsroom,-http://spie.org/x261 18. xml. 
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Summary  abstract: 

The  psychology  of  remote-operator  environmental  awareness  is  an  important  area  of  inquiry  in  which  recent 
advances  enable  improved  deployment  of  robotic  vehicles. 

The  use  of  unmanned  ground  vehicles  (UGVs)  allows  for  remote  sensing  and  action  in  hazardous  environments  at  a 
safe  distance.  Proponents  of  UGV  deployment  for  both  military  and  civilian  applications  must  overcome  technical 
challenges  faced  in  realizing  their  full  potential.  The  development  of  operationally  effective  UGVs  remains 
demanding  due  to  the  complexity  of  autonomous  ground  navigation  and  the  cognitive  requirements  on  human 
operators.  These  difficulties  appear  across  a  variety  of  UGV  applications,  including  the  use  of  large  autonomous 
vehicles  for  reconnaissance  or  convoy  operations  and  the  deployment  of  small  remotely  operated  vehicles  for 
explosives  disposal  or  urban  search-and-rescue  tasks.  1 


Chadwick,  R.  A.  (2005).  The  Impacts  of  Multiple  Robots  and  Display  Views  in  an  Urban  Search  and  Rescue 
Simulation.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  use  of  remote  controlled  uninhabited  ground  vehicles  (UGVs)  is  expanding  in  military  and  emergency 
operations.  In  operations  involving  multiple  UGVs  controlled  by  a  single  operator,  understanding  the  psychological 
implications  for  operational  proficiency  with  the  integration  of  multiple  displayed  viewpoints  is  critical.  The  current 
empirical  study  addresses  this  issue  in  an  experimental  simulation  of  an  urban  search  and  rescue  (USAR)  operation. 
Using  miniature  vehicles  in  a  simulated  environment,  participants  were  tasked  with  searching  through  difficult 
terrain  to  photograph  specific  targets.  In  the  experiment,  the  number  of  UGVs  used  and  the  use  of  an  uninhabited  air 
vehicle  (UAV)  view  were  manipulated  as  between  participants  variables.  Results  indicate:  (1)  that  the  use  of  two 
UGVs  (in  a  team  mode)  is  rather  inefficient,  (2)  that  using  two  UGVs  did  not  meaningfully  reduce  faults,  and  (3) 
that  the  use  of  a  UAV  view,  and  to  a  lesser  extent  two  UGVs,  improved  target  localization. 


Chadwick,  R.  A.  (2006).  Operating  Multiple  Semi-Autonomous  Robots:  Monitoring,  Responding,  Detecting.  In  In 
Proceedings  of  50th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  San  Francisco  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  concept  of  one  operator  controlling  multiple  unmanned  ground  vehicle  (UGV)  robots  is  examined  in  a 
simulation  of  a  multiple  UGV  search  task.  Semi-autonomous  robotic  UGVs  were  simulated  searching  for 
radioactive  targets  while  operators  performed  various  sub-tasks  including  monitoring  status,  responding  to  prompted 
target  decision  events,  and  detecting  contextual  errors.  Participants  executed  a  series  of  scenarios  using  1,  2,  or  4 
UGVs  simultaneously.  The  detection  of  contextual  errors  is  particularly  difficult  when  multiple  vehicles  are  being 
supported.  Specific  interface  features  designed  to  assist  with  multiple  vehicle  operations  are  discussed  including  a 
quick  video  playback  (QVP)  function. 


Chadwick,  R.  A.  (2008).  Considerations  for  the  Use  of  Aerial  Views  in  Unmanned  Ground  Vehicle  Operations.  In 
In  Proceedings  of  52nd  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  22-26  September  2008. 
New  York  City,  NY  USA.  New  York  City,  NY  USA:  HFES. 
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Key  Words: 
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Summary  abstract: 

Remote  unmanned  ground  vehicle  (UGV)  operations  place  the  human  operator  at  a  perceptual  disadvantage.  Adding 
aerial  views  can  benefit  the  operator’s  spatial  cognition  by  supplying  the  missing  contextual  information  regarding 
the  vehicle’s  position  and  relation  to  other  objects  in  the  space  surrounding  the  vehicle.  In  order  to  benefit  from  this 
additional  information  the  operator  must  control  and  integrate  multiple  viewpoints.  In  a  series  of  experiments  the  use 
of  aerial  views  was  examined  including  control  mode  options  and  altitude  for  the  aerial  scene  imaging.  Results 
indicate  that  aerial  views  are  beneficial  in  UGV  search  tasks  and  that  auto-tracking  aerial  imaging  control  modes 
should  be  considered. 


Chadwick,  R.  A.,  &  Gillan,  D.  J.  (2007).  Strategies  for  the  Interpretive  Integration  of  Ground  and  Air  Views  in  UGV 
Operations.  In  In  Proceedings  of  25th  Army  Science  Conference,  Orlando,  FL. 

Key  Words: 
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Summary  abstract: 

Two  experiments  examined  the  cognitive  process  of  aerial  view  target  localization.  Participants  were  shown  ground- 
view  images  with  designated  targets,  and  tasked  with  locating  the  target  in  an  aerial  view.  The  first  study  examined 
photographic  image  sets  in  both  a  qualitative  and  quantitative  manner,  including  a  think-aloud  protocol  analysis.  The 
second  study  used  manipulated  three  dimensional  model  images  to  isolate  effects  of  color,  shape,  and  other 
attributes.  Results  show  a  strong  cue  dominance  effect  for  unique  colors,  sex  differences,  and  minor  view  angle 
effects.  We  discuss  a  proposed  cognitive  model  for  this  task  and  suggest  recommendations  for  assistive  unmanned 
ground  vehicle  (UGV)  interface  features. 


Chadwick,  R.  A.,  &  Gillan,  D.  J.  (2008).  Cognitive  Integration  of  Aerial  and  Ground  Views  in  Remote  Vehicle 
Operations.  In  In  Proceedings  of  SPIE  Symposium  on  SPIE  Defense  and  Security  Symposium,  Unmanned  Systems 
Technology,  Orlando,  FL. 

Key  Words: 
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Summary  abstract: 

Both  unmanned  air  vehicles  (UAVs)  and  unmanned  ground  vehicles  (UGVs)  are  being  used  in  increasingly  complex 
roles  as  this  technology  matures.  There  are  many  proposals  for  the  joint  use  of  UAVs  and  UGVs  working  together  to 
achieve  mission  goals.  Due  to  the  spatial  perception  difficulties  commonly  reported  in  UGV  operations,  it  has  been 
postulated  that  the  operation  of  UGVs  could  benefit  from  the  use  of  live  aerial  views  in  scenarios  where  this  is 
possible.  In  a  series  of  experimental  studies  we  examine  the  cognitive  task  of  integrating  air  and  ground  views. 
Integrating  these  views  is  a  key  cognitive  task  component  in  UGV  operations  which  usually  require  that  maps  for 
navigation  be  integrated  with  imagery  from  ground  view  cameras.  Whether  aerial  views  are  used  or  not,  integration 
with  map  views  is  crucial.  In  this  paper  we  discuss  a  series  of  experimental  studies  relevant  to  the  cognitive 
integration  of  air  and  ground  views  in  UGV  scenarios.  The  integration  of  map  and  ground  camera  views  may  be 
facilitated  by  the  use  of  live  aerial  views.  In  urban  search  scenarios  the  addition  of  an  aerial  view  to  a  UGV 
operator’s  task  does  not  appear  to  have  any  measurable  negative  consequences  and  improves  localization 
performance. 


Chadwick,  R.  A.,  Gillan,  D.  J.,  Simon,  D.,  &  Pazuchanics,  S.  (2004).  Cognitive  Analysis  Methods  for  Control  of 
Multiple  Robots:  Robotics  on  $5  a  Day.  In  In  Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  New  Orleans,  LA.  (pp.  688-692). 
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Summary  abstract: 

Studies  of  a  single  operator  controlling  multiple  robots  were  designed  (1)  to  identify  problems  caused  by  perceptual 
and  cognitive  factors  and  (2)  to  propose  interface  concepts  to  reduce  the  problems.  The  studies  used  innovative 
techniques  at  extremely  low  cost.  Existing  artifacts,  i.e.,  commercially-available  video  games  played  over  local  area 
networks,  provided  a  rich  source  of  data  on  control  of  one  or  multiple  semiautonomous  entities  with  a  minimum  of 
development.  Psychological  theories  regarding  various  attributes  of  robotic  operation,  including  perception, 
navigation,  motion  control,  and  information  display  are  embedded  in  such  artifacts.  Custom  scenarios  involving 
multiple  robots  controlled  by  a  single  operator  were  developed  to  facilitate  switching  between  tasks  and,  in  some 
cases,  cooperative  use  of  the  multiple  robots.  Findings  include  data  on  navigation  difficulty,  task  switching 
strategies,  mode  errors,  and  perceptual  confusion  regarding  multiple  viewpoints. 


Chadwick,  R.  A.,  &  Pazuchanics,  S.  (2007).  Spatial  Disorientation  in  Remote  Ground  Vehicle  Operations:  Target 
Localization  Errors.  In  In  Proceedings  of  51st  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society. 
Baltimore,  MD. 

Key  Words: 
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Summary  abstract: 

Operating  remote  unmanned  ground  vehicles  (UGVs)  poses  the  risk  of  operator  spatial  disorientation.  While  the 
position  of  the  vehicle  can  be  tracked  and  displayed  on  a  global  map,  operators  face  the  difficult  task  of  integrating 
object  (target)  information  from  context  limited  ground  views  with  a  global  map  view.  Studies  indicate  this  can  be  a 
challenging  task.  An  analysis  of  specific  object  location  errors  is  presented. 


Chadwick,  R.  A.,  Pazuchanics,  S.  L.,  &  Gillan,  D.  J.  (2005).  Chapter  26.  What  the  Robot's  Camera  Tells  the 
Operator's  Brain.  In  N.  Cooke  and  H.  Pedersen  (Eds.),  Human  Factors  of  Remotely  Piloted  Vehicles:  Advances  in 
Human  Performance  and  Cognitive  Engineering  Research  (pp.  373-384).  Amsterdam:  Elsevier. 
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Summary  abstract: 

bThe  failure  of  all  15  robotic  entries  at  the  2004  DARPA  Grand  Challenge  race  across  the  Mojave  Desert  suggests 
that,  for  now,  the  possibility  of  fully  autonomous  robots  is  remote.  Among  the  most  natural  and  easy-to-do  activities 
for  mobile,  adult  humans  is  moving  through  space.  However,  that  ease  and  naturalness  belies  the  complexity  of  the 
perceptual,  cognitive,  and  motor  processes  that  underlie  visually  guided  motion.  Although  the  ease  of  human 
navigation  may  deceive  robotics  developers  into  believing  that  automating  spatial  navigation  should  be  similarly 
straightforward,  reliable  solutions  for  autonomous  navigation  and  localization  in  dynamic  real  world  environments 
have  not  been  found  (Murphy  et  al.,  1993;  Santos,  Castro,  &  Ribeiro,  2000;  Tomatis,  Nourbakhsh,  &  Siegwart, 
2003).  Furthermore,  autonomous  navigation  is  the  only  one  component  of  effective,  reasoned  autonomous  operation. 
Given  that  teleoperations  are  likely  to  continue  to  be  an  important  mode  of  robotic  control  for  the  foreseeable  future, 
what  perceptual  information  does  the  operator  of  a  ground-based  remotely  operated  vehicle  (ROV)  or  of  multiple 
ROVs  need  in  order  to  navigate  the  system  through  space,  and  how  should  the  information  be  presented  to  the 
operator? 

This  chapter  will  discuss  observational  studies  and  experiments  performed  in  the  Robotics  Control  Laboratory  in  the 
Psychology  Department  at  New  Mexico  State  University  investigating  the  perceptual  and  cognitive  processes 
involved  when  an  operator  controls  two  ROVs  simultaneously.  The  research  includes  studies  of  operators 
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controlling  virtual  ROVs  (e.g.,  using  CeeBot,  a  software -based  programmable  robot  simulation  created  by  Epistec) 
or  controlling  minirobots  in  a  miniature  environment  that,  when  seen  through  a  video-based  user  interface,  appears 
to  be  full  sized. 

The  research  that  we  summarize  in  this  chapter  includes  (1)  experiments  on  navigation  using  an  egocentric  or  first 
person  viewpoint,  including  the  role  of  landmarks  and  toe  use  of  maps,  (2)  studies  of  the  size  of  the  ROV's  field  of 
view  and  camera  viewpoint,  and  (3)  research  on  how  operators  integrate  interface  components  that  display  spatial 
position  information  with  camera-based  views  of  the  robots'  positions  to  develop  an  awareness  of  the  location  of 
multiple  ROVs.  The  chapter  will  also  discuss  the  implications  of  integrating  visual  information  provided  to  the 
operator  via  multiple  ROVs,  including  both  ground  and  air  vehicles. 


Chandrasekaran,  B.  (2002).  Intelligent  Systems  as  Control  Systems:  A  Knowledge  Level  Perspective.  In  J.H.S. 
Azuela,  C.A.  Ibanez,  M.A.  Mentado,  &  A.  Gelbukh  (Eds.),  Avances  En  Ciencias  De  La  Computacion  E  Ingenieria 
De  Computo.  Centro  De  Investigation  En  Computacion-Ipn.  Keynote  Presentation,  XI  International  Conference  on 
Computing,  Mexico  City,  November  25  to  29,  2002  (pp.  75-102):  ISBN  970-18-8590-2. 
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Intelligent  control,  knowledge  level  analysis,  intelligence  as 
control. 

Summary  abstract: 

Intelligence,  and  intelligent  systems,  can  be  viewed  from  many  perspectives:  Turing  computation,  reasoning 
systems,  search  engines,  dynamical  systems,  selectionist  structures,  and  so  on.  In  this  paper,  I  elaborate  a  view  of 
them  as  control  systems  -  a  view  intended  to  be  complementary,  not  opposed,  to  the  other  perspectives.  A  point  of 
intelligent  behavior  is  to  achieve  goals  in  complex  environments,  where  the  goals  themselves  are  generally  defined 
in  terms  of  states  of  the  environment.  Thus  behaving  intelligently  is  a  matter  of  organizing  actions  on  the 
environment  such  that  it  is  in  desirable  states  and  not  in  undesirable  states.  This  is  quintessentially  the  problem  of 
control.  What  is  it  that  unifies  the  control  task  in  all  its  manifestations,  from  the  thermostat  to  the  operator  of  a 
nuclear  power  plant?  How  do  we  explain  the  variety  of  the  solutions  that  we  see  for  the  task?  I  propose  a  Knowledge 
Level  analysis  of  the  task.  Differences  in  availability  of  knowledge,  the  degree  of  compilation  in  the  knowledge  to 
map  from  observations  to  actions,  and  properties  required  of  the  solutions  together  determine  the  differences  in  the 
solution  architectures.  I  propose  a  number  of  heuristics  that  arise  out  of  the  Knowledge  Level  analysis  for  the  design 
of  systems  to  control  the  physical  world. 


Chandrasekaran,  B.  (2002).  Multimodal  Representations  as  the  Basis  for  Cognitive  Architecture.  In  In  Proceedings 
of  ACT-R  2002  Workshop.  Carnegie  Mellon  University.  Pittsburgh,  PA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

•  Experiencing,  problem  solving,  reasoning,  takes  place  in  the  context  of  an  external  world  that  we  perceive,  act  on, 
imagine  and  reason  about  in  multiple  modalities 

-  Conceptual  modality  is  always  present 

-  Different  tasks  emphasize  different  perceptual  modalities 
•  Music  composition  vs  mechanical  design 

•  In  our  framework,  the  internal  representational  life  of  the  agent  is  multimodal,  with  representations  in  one  mode 
evoking  allied  representations  in  other  modes,  and  each  mode  making  inferential  contributions  for  which  it  is  best 
suited.  Mental  images  come  in  all  modalities. 
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•  Treats  conceptual  component  as  just  another  component  with  equal  status  with  inner  perceptual  and  kinesthetic 
components.  In  one  way  of  thinking,  having  concepts  is  imaging  the  world  in  the  conceptual  world,  just  as  having 
images  is  imaging  the  world  in  the  perceptual  mode. 

•  Logical  rules  of  inference  are  just  a  very  small  part  of  the  information  extraction  operators  in  the  conceptual  part. 


Chandrasekaran,  B.  (2002).  Reach  Exceeds  Grasp:  Comments  on  Frawley's  'Control  and  Cross-Domain  Mental 
Computation:  Evidence  from  Language  Breakdown'.  Computational  Intelligence,  18(1),  43-46. 
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Summary  abstract: 

A  particular  version  of  the  modular  theory  of  cognition  goes  like  this.  Cognition  consists  of  modules,  each  with  its 
own  domain  of  expertise.  In  this  picture,  a  module  does  not  interact  directly  with  another  module.  A  module  reports 
the  results  of  its  activity  to  a  central  working  memory  (WM),  with  its  own  more  or  less  complex  structure.  Tasks 
often  require  involvement  of  more  than  one  module,  thus  setting  up  the  problem  of  how  the  different  modules  work 
together.  This  problem  is  solved  by  having  the  modules  not  only  report  their  results  to  the  WM,  but  also  by  having 
the  modules  that  need  information  from  another  module  simply  read  off  the  WM  the  results  reported  by  the  latter 
module.  This  way,  the  task  gets  accomplished  without  the  need  for  the  modules  to  directly  communicate  with  other 
modules.  The  WM  plays  the  role  of  a  central  blackboard  in  this  view. 


Chandrasekaran,  B.  (2002).  What  Does  It  Mean  for  a  Computer  to  Do  Diagrammatic  Reasoning?:  A  Functional 
Characterization  of  Diagrammatic  Reasoning  and  Its  Implications.  In  In  Proceedings  of  Diagrams  2002  International 
Conference.  Callaway  Gardens,  GA. 

Key  Words: 
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Summary  abstract: 

What  is  Diagrammatic  Reasoning? 

Consider  some  examples: 

1.  Proving  syllogisms  using  Euler  Diagrams:  1  +  3+  5  +. .  .(2n-l)  =  n2 

For  these  problems,  consider  the  diagram  drawn  on  paper  vs  imagining  vs  a  computer  program  doing  it. 

(i)  Concrete  version  vs  the  infinite  version 

(ii)  Subtle  interplay  of  what  perception  gives  and  how  it  is  combined  with  conceptual  information 


Chandrasekaran,  B.  (2003).  Design  Problem  Solving:  Strands  of  My  Research.  In  U.  Lindemann  (Ed.),  Human 
Behavior  in  Design:  Individuals,  Teams,  Tools  (pp.  219-229).  Heidelberg:  Springer  Verlag. 
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Summary  abstract: 

My  main  research  interest  is  cognitive  architecture  for  intelligent  agents.  I  have  always  driven  my  research  by 
focusing  on  specific  tasks  that  bring  out  the  nature  of  thinking,  and  design  problem  solving  has  been  one  such  task 
area.  I  have  sought  to  identify  what  is  common  to  all  design  activity,  whether  it  is  design  of  mechanical  artifacts, 
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electronics  systems,  or  software.  What  unifies  all  of  my  work  on  design  is  my  interest  in  logical,  ontological  and 
computational  bases  of  the  generic  design  task.  My  work  on  design  can  be  thought  of  in  terms  of  strands  of  research 
that  relate  to  one  another. 


Chandrasekaran,  B.,  Kurup,  U.,  Banerjee,  B.,  Josephson,  J.,  &  Winkler,  R.  (2005).  An  Architecture  for  Problem 
Solving  with  Diagrams.  In  A.  Blackwell,  K.  Marriott,  &  A.  Shimojima  (Eds.),  Diagrammatic  Reasoning  and 
Inference,  Lecture  Notes  in  Artificial  Intelligence.  Pp.  151-165.=.  2980,  Berlin:  Springer  Verlag. 
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Summary  abstract: 

In  problem  solving  a  goal/subgoal  is  either  solved  by  generating  needed  information  from  current  information,  or 
further  decomposed  into  additional  subgoals.  In  traditional  problem  solving,  goals,  knowledge,  and  problem  states 
are  all  modeled  as  expressions  composed  of  symbolic  predicates,  and  information  generation  is  modeled  as  rule 
application  based  on  matching  of  symbols.  In  problem  solving  with  diagrams  on  the  other  hand,  an  additional  means 
of  generating  information  is  available,  viz.,  by  visual  perception  on  diagrams.  A  subgoal  is  solved  opportunistically 
by  whichever  way  of  generating  information  is  successful.  Diagrams  are  especially  effective  because  certain  types 
of  information  that  is  entailed  by  given  information  is  explicitly  available  -  as  emergent  objects  and  emergent 
relations  -  for  pickup  by  visual  perception.  We  add  to  the  traditional  problem  solving  architecture  a  component  for 
representing  the  diagram  as  a  configuration  of  diagrammatic  objects  of  three  basic  types,  point,  curve  and  region;  a 
set  of  perceptual  routines  that  recognize  emergent  objects  and  evaluate  a  set  of  generic  spatial  relations  between 
objects;  and  a  set  of  action  routines  that  create  or  modify  the  diagram.  We  discuss  how  domainspecific  capabilities 
can  be  added  on  top  of  the  generic  capabilities  of  the  diagram  system.  The  working  of  the  architecture  is  illustrated 
by  means  of  an  application  scenario. 


Chandrasekaran,  B.  (2005).  Representing  Function:  Relating  Functional  Representation  and  Functional  Modeling 
Research  Streams.  Artificial  Intelligence  for  Engineering  Design,  Analysis  and  Manufacturing,  Special  Issue  on 
Engineering  Applications  of  Representations  of  Function,  19,  65-74. 
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Device  Ontology;  Device  Structure',  Functional  Modeling;  Functional  Representation:  How  Things  Work 
Summary  abstract: 

This  paper  is  an  informal  description  of  some  recent  insights  about  what  a  device  function  is,  how  it  arises  in 
response  to  needs,  and  how  function  arises  from  the  structure  of  a  device  and  the  functions  of  its  components.  These 
results  formalize  and  darify  a  set  of  contending  intuitions  about  function  that  researchers  have  had.  The  paper  relates 
the  approachs,  results  and  goals  of  this  stream  of  research  called  functional  representation  (FR),  with  the  functional 
modeling  (FM)  stream  in  engineering. 


Chandrasekaran,  B.  (2006).  Diagrams  as  Physical  Models  to  Assist  in  Reasoning.  In  L.  Magnani  (Ed.),  Model  Based 
Reasoning  in  Science  and  Engineering,  Cognitive  Science  and  Logic.  Texts  in  Logic  Series  (pp.  1-16).  London: 
King's  College  Publications.  ISBN  1-904987-23-0. 
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Summary  abstract: 

abstract.  Diagrams  serve  a  variety  of  roles  in  applied  reasoning.  At  times,  they  are  best  viewed  as  \sentences"  in  a  2- 
D  language,  with  specialized  rules  of  inference  that  provide  new  diagrams.  Other  times,  however,  they  are  best 
understood  as  providing  a  physical  model,  much  like  an  architectural  model  of  a  building  or  a  3-D  molecular  model 
of  a  chemical  compound,  for  a  state  of  a®airs.  In  this  paper,  I  discuss  the  notion  of  a  physical  model  for  a  logical 
sentence,  and  the  role  played  by  the  causal  structure  of  the  physical  medium  in  making  the  given  sentence  as  well  as 
a  set  of  implied  sentences  true.  When  the  physical  model  is  prototypical,  it  supports  the  inference  of  certain  other 
sentences  for  which  it  provides  a  model  as  well.  I  also  informally  discuss  a  proposal  that  diagrams  and  similar 
physical  models  help  to  explicate  a  certain  sense  of  relevance  in  inference,  an  intuition  that  so-called  Relevance 
Logics  attempt  to  capture. 


Chandrasekaran,  B.  (2006).  Diagrams  as  Physical  Models.  In  D.  Barker-Plummer,  R.  Cox,  &  N.  Swoboda  (Eds.),  In 
Proceedings  of  Diagrams  2006,  the  International  Conference  on  Diagrammatic  Representations  and  Reasoning  at 
Stanford  University  (pp.  204-217):  Springer. 
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Summary  abstract: 

focusing  in  particular  on  their  role  as  physical  models  of  states  of  affairs,  much  like  an  architectural  model  of  a 
building  or  a  3-D  molecular  model  of  a  chemical  compound.  We  discuss  the  concept  of  a  physical  model  for  a 
logical  sentence,  and  the  role  played  by  the  causal  structure  of  the  physical  medium  in  making  the  given  sentence  as 
well  as  a  set  of  implied  sentences  true.  This  role  of  a  diagram  is  consistent  with  a  widely-held  intuition  that  diagrams 
exploit  the  fact  that  2-D  space  is  an  analog  of  the  domain  of  discourse.  One  line  of  research  in  diagrammatic 
reasoning  is  that  diagrams,  rather  then  being  models,  are  formal  representations  with  specialized  rules  of  inference 
that  generate  new  diagrams.  We  reconcile  these  contrasting  views  by  relating  the  usefulness  of  diagrammatic 
systems  as  formal  representations  to  the  fact  that  their  rewrite  rules  take  advantage  of  the  diagrams’  model-like 
character.  When  the  physical  model  is  prototypical,  it  supports  the  inference  of  certain  other  sentences  for  which  it 
provides  a  model  as  well.  We  also  informally  discuss  a  proposal  that  diagrams  and  similar  physical  models  help  to 
explicate  a  certain  sense  of  relevance  in  inference,  an  intuition  that  so-called  Relevance  Logics  attempt  to  capture. 


Chandrasekaran,  B.  (2006).  Multimodal  Cognitive  Architecture:  Making  Perception  &  Body  More  Central  to 
Intelligent  Behavior.  In  In  Proceedings  of  the  AAAI  National  Conference  on  Artificial  Intelligence,  (www.aaai.org) 
(pp.  1508-1512.). 
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Summary  abstract: 

I  propose  that  the  notion  of  cognitive  state  be  broadened  from  the  current  predicate-symbolic,  Language-of-Thought 
framework  to  a  multi-modal  one,  where  perception  and  kinesthetic  modalities  participate  in  thinking.  In  contrast  to 
the  roles  assigned  to  perception  and  motor  activities  as  modules  external  to  central  cognition  in  the  currently 
dominant  theories  in  AI  and  Cognitive  Science,  in  the  proposed  approach,  central  cognition  incorporates  parts  of  the 
perceptual  machinery.  I  motivate  and  describe  the  proposal  schematically,  and  describe  the  implementation  of  a  bi- 
modal  version  in  which  a  diagrammatic  representation  component  is  added  to  the  cognitive  state.  The  proposal 
explains  our  rich  multimodal  internal  experience,  and  can  be  a  key  step  in  the  realization  of  embodied  agents.  The 
proposed  multimodal  cognitive  state  can  significantly  enhance  the  agent’s  problem  solving. 
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Chandrasekaran,  B.  (2006).  Multi-Modal  Cognitive  States:  Augmenting  the  State  in  Cognitive  Architectures.  In  In 
Proceedings  of  American  Association  for  Artificial  Intelligence  Spring  Symposium  Series,  Stanford,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Different  streams  of  AI  idealize  different  aspects  of  human  cognition.  Idealization  of  intelligence  as  an  embodied 
activity,  involving  an  integration  of  cognition,  perception  and  the  body,  places  the  tightest  constraints  on  the  design 
space  for  AI  artifacts,  forcing  AI  to  deeply  understand  the  design  tradeoffs  and  tricks  that  biology  has  developed.  1 
propose  that  a  step  in  the  design  of  such  artifacts  is  to  broaden  the  notion  of  cognitive  state  from  the  current 
linguistic-symbolic,  Language-of-Thought  framework  to  a  multi-modal  one,  where  perception  and  kinesthetic 
modalities  participate  in  thinking.  This  is  in  contrast  to  the  roles  assigned  to  perception  and  motor  activities  as 
modules  external  to  central  cognition  in  the  currently  dominant  theories  in  AI  and  Cognitive  Science.  I  develop  the 
outlines  of  this  proposal,  and  describe  the  implementation  of  a  bimodal  version  in  which  a  diagrammatic 
representation  component  is  added  to  the  cognitive  state. 


Chandrasekaran,  B.  (2008).  Designing  Decision  Support  Systems  to  Help  Avoid  Biases  &  Make  Robust  Decisions, 
with  Examples  from  Army  Planning.  In  In  Proceedings  of  2008  Army  Science  Conference. 
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Summary  abstract: 

This  paper  is  concerned  with  two  sets  of  issues  related  to  optimality  in  planning.  The  first  is  a  proposal  that  design 
of  decision  support  systems  (DSS’s)  for  planning  should  aim  to  support  the  planner  in  generating  a  plan  that  is 
robust,  i.e.,  has  satisfactory  performance  even  when  reality  differs  from  assumptions.  Such  a  plan  would  sacrifice 
optimality  when  reality  is  as  assumed  for  reasonable  performance  over  a  larger  range  of  situations.  We  discuss  how 
this  proposed  refocus  follows  from  the  in-principle  incompleteness,  and  common  errorfulness,  of  domain  models 
required  to  assess  the  performance  of  plans.  The  second  issue  related  to  optimality  is  the  degree  to  which  human 
judgment  in  planning  is  subject  to  a  number  of  biases,  all  detracting  from  optimality.  The  Framing  Bias  arises  from 
the  Bounded  Rationality  of  human  cognition.  The  Transitivity  Bias  is  a  result  of  treating  small  and  large  differences 
in  the  criteria  values  as  of  equal  importance.  Our  analysis  leads  to  design  recommendations  for  DSS's  that  provide  a 
measure  of  protection  against  these  biases.  The  ideas  are  motivated  and  illustrated  with  Army  planning  examples. 


Chandrasekaran,  B.  (2009).  Problem  Solving  Methods  and  Knowledge  Systems:  A  Personal  Journey  to  Perceptual 
Images  as  Knowledge.  Artificial  Engineering  in  Engineering  Design  and  Manufacturing,  Special  Issue  on  Problem 
Solving  Methods:  Past,  Present  and  Future,  2009,  Vol.  23,  No.  3. 
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Summary  abstract: 

I  was  among  those  who  proposed  problem  solving  methods  (PSMs)  in  the  late  1970s  and  early  1980s  as  a 
knowledge-level  description  of  strategies  useful  in  building  knowledge-based  systems.  This  paper  summarizes  the 
evolution  of  my  ideas  in  the  last  two  decades.  I  start  with  a  review  of  the  original  ideas.  From  an  artificial 
intelligence  (AI)  point  of  view,  it  is  not  PSMs  as  such,  which  are  essentially  high-level  design  strategies  for 
computation,  that  are  interesting,  but  PSMs  associated  with  tasks  that  have  a  relation  to  AI  and  cognition.  They  are 
also  interesting  with  respect  to  cognitive  architecture  proposals  such  as  Soar  and  ACT-R:  PSMs  are  observed 
regularities  in  the  use  of  knowledge  that  an  exclusive  focus  on  the  architecture  level  might  miss,  the  latter  providing 
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no  vocabulary  to  talk  about  these  regularities.  PSMs  in  the  original  conception  are  closely  connected  to  a  specific 
view  of  knowledge:  symbolic  expressions  represented  in  a  repository  and  retrieved  as  needed.  I  join  critics  of  this 
view,  and  maintain  with  them  that  most  often  knowledge  is  not  retrieved  from  a  base  as  much  as  constructed  as 
needed.  This  criticism,  however,  raises  the  question  of  what  is  in  memory  that  is  not  knowledge  as  traditionally 
conceived  in  AI,  but  can  support  the  construction  of  knowledge  in  predicate-symbolic  form.  My  recent  proposal 
about  cognition  and  multimodality  offers  a  possible  answer.  In  this  view,  much  of  memory  consists  of  perceptual 
and  kinesthetic  images,  which  can  be  recalled  during  deliberation  and  from  which  internal  perception  can  generate 
linguistic-  symbolic  knowledge.  For  example,  from  a  mental  image  of  a  configuration  of  objects,  numerous 
sentences  can  be  constructed  describing  spatial  relations  between  the  objects.  My  work  on  diagrammatic  reasoning 
is  an  implemented  example  of  how  this  might  work.  These  internal  perceptions  on  imagistic  representations  are  a 
new  kind  of  PSM. 


Chandrasekaran,  B.,  Banerjee,  B.,  Kurap,  U.,  Josephson,  J.  R.,  &  Winkler,  R.  (2009).  Chapter  21.  Diagrammatic 
Reasoning  in  Army  Situation  Understanding  and  Planning:  Architecture  for  Decision  Support  and  Cognitive 
Modeling.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter: 
Foundations  and  Technology.  Section  IV.  Communicating  Information  across  the  Team:  48hrBooks  by  Alion 
Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving 
Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision 
Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board 
Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue 
SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 
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Summary  abstract: 

Diagrams  are  ubiquitous  in  army  decision-making.  Commanders  represent,  communicate,  and  monitor  their 
situation  understanding  and  plans  by  means  of  diagrams  drawn  over  terrain  or  other  maps.  Diagrams  are  so  central 
in  military  applications  that  field  manuals  have  standardized  the  elements  of  such  representations.  They  are  not  only 
useful  in  spatial  reasoning  tasks,  but  also  in  tasks  that  are  not  directly  related  to  space,  as  evidenced  by  bar  and  pie 
charts  and  Venn  diagrams.  The  reason  for  their  usefulness  is  that  they  enable  an  optimal  combination  of  visual 
perception  and  symbolic  reasoning:  visual  perception  picks  up  information  that  is  perceptually  available  in  the 
diagram,  avoiding  chains  of  inference,  while  symbolic  reasoning  enables  inferences  that  can  only  be  made  with 
abstract  and  distal  knowledge. 

We  have  two  different  sets  of  motivations  for  research  in  diagrammatic  representations,  and  both  are  especially 
relevant  to  army  decision  support.  The  first  is  the  need  to  understand  what  makes  for  good  diagrams  for  a  task,  i.e., 
what  graphical  properties  of  a  diagram  make  it  easier  for  a  human  user  to  obtain  the  needed  information  fast  and 
without  error.  This  knowledge  can  be  used  to  design  effective  diagrams  and  diagrammatic  interfaces  for  interactive 
systems.  Exploring  the  interaction  between  task  demands,  graphical  properties,  human  cognitive  and  perceptual 
architecture,  and  the  user's  background  knowledge  that  go  into  using  a  diagram  is  best  done  by  building  cognitive 
models.  Designing  and  building  a  cognitive  architecture  that  rises  to  the  challenge  of  modeling  diagrammatic 
reasoning  has  been  a  major  concern  of  ourresearch. 

The  second  motivation  arises  from  the  opportunity  to  enhance  decision-  making  by  automating  some  ofthe  reasoning 
tasks.  For  example,  consider  replanning  a  route  in  real  time  in  an  urban  engagement  in  response  to  rapid  changes  in 
the  situation,  especially  if  the  changes  are.  occurring  outside  of  the  immediate  surroundings  of  a  soldier.  For  a 
soldier  in  the  heat  of  the  action,  this  task  will  be  very  demanding  at  best  and  impossible  at  worst.  An  automated 
diagrammatic  reasoning  system  that  can  perform  the  needed  rerouting  fast  and  display  the  results  in  a  form  that  the 
soldier  can  comprehend  can  be  very  helpful.  As  another  example,  a  planner  might  desire  to  critique  a  planned 
avenue  of  approach  with  respect  to  risk  of  potential  ambush.  The  complexity  of  the  terrain  may  make  the  task  time- 
consuming,  but  an  automated  diagrammatic  reasoning  system  that  can  perform  this  task,  as  a  demonstration  system 
we  have  built  does,  can  add  real  value.  Both  the  research  goalsautomation  and  cognitive  modeling-  have  a  common 
element:  a  general  cognitive  architecture  that  can  perform  diagrammatic  reasoning  by  appropriately  combining 
spatial  information  and  abstract  knowledge. 


112 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 

Chandrasekaran,  B.,  &  Goldman,  M.  (2007).  Exploring  Robustness  of  Plans  for  Simulation-Based  Course  of  Action 
Planning:  A  Framework  and  an  Example.  In  In  Proceedings  of  IEEE  Symposium  on  Computational  Intelligence  in 
Multi-Criteria  Decision-Making  (MCDM  2007),  (http://www.ieeecis-multidecision.org/)  Honolulu,  Hawaii. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Planning  requires  evaluating  candidate  plans  multicriterially,  which  in  turn  requires  some  kind  of  a  causal  model  of 
the  operational  environment,  whether  the  model  is  to  be  used  as  part  of  evaluation  by  humans  or  simulation  by 
computers.  However,  there  is  always  a  gap  -  consisting  of  missing  or  erroneous  information  -  between  any  model 
and  the  reality.  One  of  the  important  sources  of  gaps  in  models  is  built-in  assumptions  about  the  world,  e.g.,  enemy 
capabilities  or  intent  in  military  planning.  Some  of  the  gaps  can  be  handled  by  standard  approaches  to  uncertainty, 
such  as  optimizing  expected  values  of  the  criteria  of  interest  based  on  assumed  probability  distributions.  However, 
there  are  many  problems,  such  as  military  planning,  where  it  is  not  appropriate  to  choose  the  best  plan  based  on  such 
expected  values,  or  where  meaningful  probability  distributions  are  not  available.  Such  uncertainties,  often  called 
"deep  uncertainties,"  require  an  approach  to  planning  where  the  task  is  not  choosing  the  optimal  plan  as  much  as  a 
robust  plan,  one  that  would  do  well  enough  even  in  the  presence  of  such  uncertainties.  Decision  support  systems 
should  help  the  planner  explore  the  robustness  of  candidate  plans.  In  this  paper,  we  illustrate  this  functionality, 
robustness  exploration,  in  the  domain  of  network  disruption  planning,  an  example  of  effect-based  operations. 


Chandrasekaran,  B.,  Josephson,  J.  R.,  Banerjee,  B.,  Kurup,  U.,  &  Winkler,  R.  (2003).  Diagrammatic  Reasoning  in 
Support  of  Situation  Understanding  and  Planning.  In  In  Proceedings  of  Collaborative  Technology  Alliances 
Conference  2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University 
of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

Visual  representations  consisting  of  terrain  maps  with  an  overlay  of  diagrammatic  elements  are  ubiquitous  in  Army 
situation  understanding,  planning  and  plan  monitoring.  The  Objective  Force  requirements  of  responsiveness,  agility 
and  versatility  call  for  digitized  graphical  decision  support  interfaces  that  automate  or  otherwise  help  in  various 
reasoning  tasks.  The  main  purpose  of  the  paper  is  to  describe  the  issues  involved  in  building  a  diagrammatic 
reasoning  system,  specifications  for  diagrammatic  representation  formalism,  and  an  architecture  for  problem  solving 
with  diagrams.  We  report  on  the  current  state  of  implementation  of  an  application  that  infers  maneuvers  from  data 
about  locations  and  motions  of  Blue  and  Red  forces. 


Chandrasekaran,  B.,  Josephson,  J.  R.,  Banerjee,  B.,  Kurup,  U.,  &  Winkler,  R.  (2003).  Diagrammatic  Reasoning  in 
Support  of  Situation  Understanding  and  Planning.  In  In  Proceedings  of  Army  Science  Conference.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Visual  representations  consisting  of  terrain  maps  with  an  overlay  of  diagrammatic  elements  are  ubiquitous  in  Army 
situation  understanding,  planning  and  plan  monitoring.  The  Objective  Force  requirements  of  responsiveness,  agility 
and  versatility  call  for  digitized  graphical  decision  support  interfaces  that  automate  or  otherwise  help  in  various 
reasoning  tasks.  The  main  purpose  of  the  paper  is  to  describe  the  issues  involved  in  building  a  diagrammatic 
reasoning  system,  specifications  for  a  diagrammatic  representation  formalism,  and  an  architecture  for  problem 
solving  with  diagrams.  We  have  begun  implementation  of  an  application  that  infers  maneuvers  from  data,  obtained 
from  an  exercise  at  the  National  Training  Center,  about  locations  and  motions  of  Blue  and  Red  forces.  We  present 
algorithms  used  in  the  initial  stages  of  an  implementation. 
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Chandrasekaran,  B.,  Josephsona,  J.,  O’May,  J.,  Heilman,  E.,  &  Kaste,  R.  (2004).  Mining  Simulation  Data  for 
Insights  About  a  Decision  Space:  Application  to  an  Urban  Combat  Coa.  In  In  Proceedings  of  SPIE,  Enabling 
Technologies  for  Simulation  Science  VIII,  Dawn  A.  Trevisani  and  Alex  F.  Sisti,  (Eds.),  Bellingham,  WA:  SPIE. 

Vol.  5423  (pp.  32-42). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
decision  architecture,  data  mining,  course  of  action,  combat  simulation,  planning,  multi-criterial  decisionmaking 

Summary  abstract: 

We  start  with  a  vision  of  an  integrated  decision  architecture  to  assist  in  the  various  stages  and  subtasks  of 
decisionmaking.  We  briefly  describe  how  the  Seeker-Filter-Viewer  (S-F-V)  architecture  for  multi-criterial  decision 
support  helps  realize  many  components  of  that  vision.  The  rest  of  the  paper  is  devoted  to  one  of  the  components: 
developing  insights  about  the  course  of  action  (COA)  decision  space  from  COA  simulations.  We  start  with  data 
obtained  from  multiple  simulation  executions  of  an  urban  combat  COA  in  a  specified  scenario,  where  the  stochastic 
nature  of  different  executions  produce  a  range  of  intermediate  events  and  final  outcomes.  The  Viewer  in  the  S-F-V 
decision  architecture  is  used  to  make  and  visually  test  hypotheses  about  how  sensitive  different  events  and  outcomes 
are  to  different  aspects  of  the  COA  and  to  various  intermediate  events.  The  analyst  engages  in  a  cycle  of  hypothesis 
making,  visually  evaluating  the  hypothesis,  and  making  further  hypotheses.  A  set  of  snapshots  illustrates  an 
investigational  sequence  of  abstractions  in  an  example  of  iterating  on  hypotheses.  The  synergy  of  data  mining  tools, 
high  performance  computing,  and  advanced  high-resolution  combat  simulation  has  the  potential  to  assist  battle 
planners  to  make  better  decisions  for  imminent  combat. 


Chandrasekaran,  B.,  &  Kurup,  U.  (2005).  Augmenting  Cognitive  State  with  Diagrams.  In  In  Proceedings  of  2005 
Soar  Workshop.  Ohio  State  University  LAIR,  Ann  Arbor,  MI. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

•  Traditional  Al/CogSci  state  representation  (Soar,  ACT-R)  is  “predicate  symbolic.” 

•  State  change  is  accomplished  by  rule-based  operators  that  transform  the  predicate  symbolic  representation. 
(Inference) 

•  Cognitive  architectures  such  as  SOAR  should  support  an  augmented  notion  of  cognitive  state 

-  Traditional  “symbolic”  will  be  one,  generally  dominant,  component  of  state,  but  other  perceptual  and 
kinesthetic  modalities  need  to  be  supported 

-  State  change  operations  get  more  complex 

-  Diagrammatic  representations  make  a  good  place  to  start  because  of  their  ubiquity  and  usefulness  in  problem 
solving 


Chandrasekaran,  B.,  &  Kurup,  U.  (2006).  A  Bimodal  Cognitive  Architecture:  Explorations  in  Architectural 
Explanation  of  Spatial  Reasoning.  In  In  Proceedings  of  American  Association  for  Artificial  Intelligence  (AAAI) 
Spring  Symposium  (www.aaai.org). 

Key  Words: 
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Summary  abstract: 

Research  in  Psychology  often  involves  the  building  of  computational  models  to  test  out  various  theories.  The  usual 
approach  is  to  build  models  using  the  most  convenient  tool  available.  Newell  has  instead  proposed  building  models 
within  the  framework  of  general-purpose  cognitive  architectures.  One  advantage  of  this  approach  is  that  in  some 
cases  it  is  possible  to  provide  more  perspicuous  explanations  of  experimental  results  in  different  but  related  tasks,  as 
emerging  from  an  underlying  architecture.  In  this  paper,  we  propose  the  use  of  a  bimodal  cognitive  architecture 
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called  biSoar  in  modeling  phenomena  in  spatial  representation  and  reasoning.  We  show  biSoar  can  provide  an 
architectural  explanation  for  the  phenomena  of  simplification  that  arises  in  experiments  associated  with  spatial 
recall.  We  build  a  biSoar  model  for  one  such  spatial  recall  task  -  wayfinding,  and  discuss  the  role  of  the  architecture 
in  the  emergence  of  simplification. 


Chandrasekaran,  B.,  Kurup,  U.,  &  Banerjee,  B.  (2005).  A  Diagrammatic  Reasoning  Architecture:  Design, 
Implementation  and  Experiments.  In  In  Proceedings  of  AAAI  Spring  Symposium,  Reasoning  with  Mental  and 
External  Diagrams:  Computational  Modeling  and  Spatial  Assistance,  Stanford  University,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  explores  the  idea  that  the  cognitive  state  during  problem  solving  diagrams  is  bi-modal,  one  of  whose 
components  is  the  traditional  predicate-symbolic  representation  composed  of  relations  between  entities  in  the 
domain  of  interest,  while  a  second  component  is  an  internal  diagrammatic  representation.  In  parallel  with  the 
operators  in  the  symbolic  representation  that  are  based  on  symbol  matching  and  inferencing,  there  is  a  set  of 
operators  in  the  diagrammatic  component  that  apply  perceptions  to  the  elements  of  the  diagram  to  generate 
information.  In  addition  there  is  a  set  of  diagram  construction  operations  that  may  modify  the  diagram  by  adding, 
deleting  and  modifying  the  diagrammatic  elements,  in  the  service  of  problem  solving  goals.  We  describe  the  design 
of  the  diagrammatic  component  of  the  architecture,  and  show  how  the  symbolic  and  diagrammatic  modes 
collaborate  in  the  solution  of  a  problem.  We  end  the  paper  with  a  view  of  the  cognitive  state  as  multi-modal,  in 
consonance  with  our  own  phenomenal  sense  of  experiencing  the  world  in  multiple  modalities  and  using  these  senses 
in  solving  problems. 


Chandrasekaran,  B.,  Kurup,  U.,  &  Banerjee,  B.  (2005).  Representational  and  Inferential  Requirements  for 
Diagrammatic  Reasoning  in  the  Entity-Reidentification  Task.  In  In  Proceedings  of  24th  Army  Science  Conference. 
Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Diagrammatic  reasoning  is  ubiquitous  in  Army  reasoning:  situation  understanding  and  planning  in  the  Army  both 
involve  representing  aspects  of  the  situation  and  plans  in  the  form  of  diagrams.  We  have  been  developing  a  general 
architecture  to  support  diagrammatic  reasoning  for  Army  applications,  and  in  an  earlier  report  [1]  we  discussed  an 
application  in  simple  maneuver  recognition.  Our  research  strategy  has  been  to  investigate  a  variety  of  applications, 
each  bringing  additional  requirements  for  perception  and  diagrammatic  object  creating  capabilities  that  can  assist  the 
army.  One  such  area  is  the  army's  All  Source  Analysis  System  (ASAS)  that  is  designed  to  automate  the  processing 
and  analysis  of  intelligence  data  from  all  possible  sources.  One  of  its  purposes  is  to  keep  track  of  various  enemy 
assets,  based  on  a  variety  of  sensors  and  reports,  including  direct  sightings  of  entities.  In  order  to  maintain  a 
coherent  view,  the  system  has  to  decide  whether  a  new  sighting  refers  to  a  new  entity,  a  previously  known  entity  that 
has  since  moved  (re-identification)  or  is  erroneous.  This  is  the  entity  re-identification  task.  We  have  a  built  a  system 
that  uses  an  abductive  reasoning  process  together  with  a  diagrammatic  reasoning  system  to  solve  this  problem.  In 
this  paper,  we  look  at  some  of  the  issues  that  the  entity  re-identification  task  poses  for  diagrammatic  reasoning. 


Chandrasekaran,  B.,  &  Yovtchev,  T.  (2007).  A  Qualitative  Goms  Approach  to  Evaluating  Diagrammatic  Interfaces. 
In  In  Proceedings  of  ICCM  -  2007-  Eighth  International  Conference  on  Cognitive  Modeling.  205  -  210.  Oxford,  UK: 
Taylor  &  Francis/Psychology  Press. 
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ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
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Summary  abstract: 

We  describe  an  approach  to  evaluating  diagrammatic  schemes  intended  to  support  problem  solving  and  decision¬ 
making.  The  methodology  is  in  the  GOMS  framework  in  HCI,  and  is  based  on  recognizing  that  the  use  of  diagrams 
is  part  of  a  process  that  can  be  decomposed  into  a  sequence  of  steps,  each  of  which  may  be  a  Perception  on  the 
diagram,  Inference,  Transformation  of  the  diagram  and  Visual  Search.  How  well  a  diagrammatic  scheme  helps  in  a 
task  depends  on  how  well  the  human  cognitive  architecture  can  perform  the  actions  in  the  various  steps,  and  how  the 
steps  collectively  contribute  to  performance  measures  such  as  time,  error  rates,  and  memory  stress.  We  illustrate  the 
approach  by  using  it  to  analyze  the  use  of  some  common  data  presentation  displays  in  the  task  of  discovering 
interesting  relations  between  variables  in  a  domain.  Because  of  the  current  lack  of  quantitative  empirical  data  about 
the  execution  of  the  basic  operations  by  human  architecture,  the  analysis  is  qualitative,  which  is  nevertheless  useful 
in  providing  useful  insights.  It  also  sets  an  agenda  for  empirical  research  to  obtain  the  quantitative  data  needed,  since 
the  availability  of  such  data  would  help  significantly  in  evaluating  and  improving  diagrammatic  interfaces  for 
decision  support. 


Chen,  D.,  Malkin,  R.,  &  Yang,  J.  (2005).  Multimodal  Detection  of  Human  Interaction  Events  in  a  Nursing  Home 
Environment.  In  In  Proceedings  of  6th  International  Conference  On  Multimodal  Interfaces.  State  College,  PA. 
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1.4.8  [Scene  analysis]:  motion,  color,  shape,  tracking,  stereo 
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Keywords 
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stochastic  modeling 

Summary  abstract: 

In  this  paper,  we  propose  a  multimodal  system  for  detecting  human  activity  and  interaction  patterns  in  a  nursing 
home.  Activities  of  groups  of  people  are  firstly  treated  as  interaction  patterns  between  any  pair  of  partners  and  are 
then  further  broken  into  individual  activities  and  behavior  events  using  a  multi-level  context  hierarchy  graph.  The 
graph  is  implemented  using  a  dynamic  Bayesian  network  to  statistically  model  the  multi-level  concepts.  We  have 
developed  a  coarse-to-fine  prototype  system  to  illustrate  the  proposed  concept.  Experimental  results  have 
demonstrated  the  feasibility  of  the  proposed  approaches.  The  objective  of  this  research  is  to  automatically  create 
concise  and  comprehensive  reports  of  activities  and  behaviors  of  patients  to  support  physicians  and  caregivers  in  a 
nursing  facility. 


Christ,  R.  E.,  Alvah  C.  Bittner,  J.,  Freeman,  J.  T.,  Archer,  R.,  Klein,  G.,  &  Dyer,  J.  L.  (2003).  Training  Rapid 
Decision-Making  Processes  Required  by  the  Dismounted  Objective  Force  Leader.  In  In  Proceedings  of  47th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  Army’s  Objective  Force  concept  exploits  the  enormous  opportunities  made  possible  by  advances  in  our  capacity 
to  quickly  gather,  organize,  and  distribute  battlespace  information.  Electronic  information  systems  will  be  developed 
and  fielded  to  process  and  display  critical  features  of  the  data  available  from  multiple  sensor  and  database  systems. 
The  Objective  Force  Warrior  is  one  component  of  the  Objective  Force  that  encompasses  a  system  of  equipment  and 
capabilities  for  the  individual  warfighter.  This  panel  session  is  concerned  directly  with  the  use  of  these  systems  by 
the  light,  dismounted  Infantryman.  The  Army  clearly  anticipates  that  innovative  uses  of  advanced  information 
systems  will  enhance  the  Infantryman’s  situational  awareness  and  his  decision-making  capabilities. 
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Christoffersen,  K.,  &  Woods,  D.  D.  (2002).  Chapter  1.  How  to  Make  Automated  Systems  Team  Players.  In  E.  Salas 
(Ed.),  Advances  in  Human  Performance  and  Cognitive  Engineering  Research  (Vol.  2,  pp.  1-12).  St.  Louis,  MO: 
Elsevier  Science. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Advances  in  technology  and  new  levels  of  automation  have  had  many  effects  in  operational  settings.  There  have 
been  positive  effects  from  both  an  economic  and  a  safety  point  of  view.  Unfortunately,  operational  experience,  field 
research,  simulation  studies,  incidents,  and  occasionally  accidents  have  shown  that  new  and  surprising  problems 
have  arisen  as  well.  Breakdowns  that  involve  the  interaction  of  operators  and  computer-based  automated  systems  are 
a  notable  and  dreadful  path  to  failure  in  these  complex  work  environments. 

Over  the  years,  Human  Factors  investigators  have  studied  many  of  the  "natural  experiments"  in  human-automation 
cooperation  -  observing  the  consequences  in  cases  where  an  organization  or  industry  shifted  levels  and  kinds  of 
automation.  One  notable  example  has  been  the  many  studies  of  the  consequences  of  new  levels  and  types  of 
automation  on  the  flight  deck  in  commercial  transport  aircraft  (from  Wiener  &  Curry,  1980  to  Billings,  1996).  These 
studies  have  traced  how  episodes  of  technology  change  have  produced  many  surprising  effects  on  many  aspects  of 
the  systems  in  question. 

New  settings  are  headed  into  the  same  terrain  (e.g.  free  flight  in  air  traffic  management,  unmanned  aerial  vehicles, 
aero-medical  evacuation,  naval  operations,  space  mission  control  centers,  medication  use  in  hospitals).  What  can  we 
offer  to  jump  start  these  cases  of  organizational  and  technological  change  from  more  than  30  years  of  investigations 
on  human-automation  cooperation  (from  supervisory  control  studies  in  the  1970s  to  intelligent  software  agents  in  the 
1990s)? 

Ironically,  despite  the  numerous  past  studies  and  attempts  to  synthesize  the  research,  a  variety  of  myths, 
misperceptions,  and  debates  continue.  Furthermore,  some  stakeholders,  aghast  at  the  apparent  implications  of  the 
research  on  humanautomation  problems,  contest  interpretations  of  the  results  and  demand  even  more  studies  to 
replicate  the  sources  of  the  problems. 


Coffey,  J.  W.,  Hoffman,  R.  R.,  Canas,  A.  J.,  &  Ford,  K.  M.  (2002).  A  Concept  Map-Based  Knowledge  Modeling 
Approach  to  Expert  Knowledge  Sharing.  In  In  Proceedings  of  International  Conference  on  Information  and 
Knowledge  Sharing. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Knowledge  Elicitation,  Knowledge  Modeling,  Knowledge  Sharing,  Concept  Maps,  PreSERVe  Method 

Summary  abstract: 

An  important  aspect  of  knowledge  management  is  the  implementation  of  methods  to  share  the  idiosyncratic 
knowledge  of  expert  practitioners  within  an  organization.  In  order  to  make  such  knowledge  sharable,  it  is  necessary 
to  have  both  an  effective  elicitation  method  and  a  useful  representation  scheme.  In  this  paper  we  describe  the 
PreSERVe  method  of  knowledge  elicitation  as  it  is  used  with  a  knowledge  representation  scheme  based  upon 
concept  maps  [1].  We  describe  the  use  of  these  methods  in  a  case  study  on  the  capture  and  representation  of  local 
weather  forecasting  knowledge. 


Cooke,  N.  J.,  &  Chadwick,  R.  A.  (2010).  Chapter  19.  Lessons  Learned  from  Human-Robotic  Interaction  on  the 
Ground  and  in  the  Air.  In  M.  Barnes  and  F.  Jentsch  (Eds.),  Human-Robot  Interactions  in  Future  Military  Operations: 
Advanced  Decision  Architectures,  Collaborative  Technology  Alliance. 

Key  Words: 

ADA  CTA 
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Summary  abstract: 

Unmanned  or,  more  specifically,  remotely-operated  vehicles  have  contributed  to  the  changing  nature  ofwarfare,  and 
have  taken  center  stage  for  civilian  applications  ranging  from  broader  security  and  monitoring  of  wild  fires,  to 
search  and  rescue,  and  in  surveillance  o(disaster  locations.  Technological  advances  now  make  it  possible  for  soldiers 
to  use  (unmanned  combat  aerial  vehicles  (UCAVs)  to  fight  the  enemy  in  the  Middle  East  from  a  trailer  in  Nevada 
(Moran  &  Morris,  2006).  It  also  makes  it  possible  for  search  efforts  to  proceed  in  places  too  difficult  for  humans  to 
enter,  such  as  the  Katrina-ravaged  neighborhoods  or  the  ruins  of  the  World  Trade  Center,  through  the  use  of  rescue 
robots  (Murphy,  2004a,  2004b).  These  robotic  vehicles  have  taken  humans  out  of  harm's  way  in  situations  known  to 
be  "dull,  dirty,  or  dangerous." 

Although  the  technology  has  progressed,  it  is  not  without  setbacks.  Remotely  operated  air  vehicles  have  a  high  crash 
rate  that  has  been  attributed  to  early  fielding  of  the  technology,  sensitivity  to  weather,  and  human  factors  concerns 
(Rash  et  al.,  2006;  Williams,  2004).  Similarly,  ground  vehicles  have  had  a  share  of  problems.  Spatial  disorientation 
is  an  inherent  and  consistently  reported  problem  in  the  operation  of  ground  vehicles  (Chadwick  &  Pazuchanics, 
2007;  Tittle  et  al.,  2002).  Poor  reliability  of  search  robots  (Carlson  &  Murphy,  2003,  2004)  and  lack  of  adequate 
vehicle  situated  awareness  (Riley  &  Endsley,  2004)  continue  to  limit  their  usefulness. 

Despite  the  presumed  role  ofhuman  factors  in  the  failure  of  these  technologies,  an  aspect  of  the  technology  that  has 
received  minimal  attention  is  the  humanrobotic  interface.  This  is  often  overlooked  in  lieu  of  a  focus  on  the  hardware 
and  a  tendency  to  "forget  about"  the  operators  who  are  located  elsewhere  (Cooke  et  al.,  2006b). 


Cox,  D.  A.,  Ross,  K.  G.,  &  Ross,  W.  A.  (2005).  Lightweight  Synthetic  Task  Environments  for  C2  Research 
Experimentation  In  In  Proceedings  of  10th  International  Command  and  Control  Research  and  Technology 
Symposium.  Qu,bec  City,  Canada. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Force  Transformation  can  be  seen  as  a  battle  against  complexity  and  uncertainty.  This  is  especially  true  when  we 
consider  the  less  concrete  components  that  are  central  to  effective  Command  and  Control  (C2).  Experimentation  is 
the  force  we  deploy  in  this  battle  as  researchers  (Alberts  &  Hayes,  2002).  Our  main  force  is  large-scale 
experimentation  and  simulation.  As  with  the  main  force  in  combat,  the  energy  of  experimentation  can  be  squandered 
if  not  properly  focused.  Large-scale  experimentation  also  has  a  large  logistical  tail,  making  it  a  precious  resource  in 
an  era  of  increasing  demands  and  restrictions.  As  such,  we  must  be  sure  that  it  is  deployed  wisely,  to  maximum 
effect.  In  this  paper  we  describe  our  lightweight  Synthetic  Task  Environment  (STE)  concept  and  how  it  fits  within 
the  experimentation  campaign  framework.  We  report  on  our  development  and  use  of  the  lightweight  STE  concept. 
We  describe  the  central  construct,  cognitive  authenticity.  We  have  set  forward  a  process  for  creating  STEs,  and  a 
framework  for  documenting  their  design.  Further  experience  is  needed  to  learn  how  to  optimize  the  creation  of  STEs 
so  that  they  can  keep  pace  with  force  transformation  efforts  in  the  US  Armed  Forces. 

STE  Development  Process 

1.  Develop  research  question 

2.  Investigate  current  or  analogous  performance 

3.  Select  features  to  preserve  or  simplify 

4.  Review  abstract  design 

5.  Initial  design  of  STE 

6.  Design  review 

7.  Complete  construction  of  STE 

8.  Pilot  test 


Cox,  D.  A.,  &  Veinott,  B.  (2009).  Developing  Pattern  Languages:  New  Ways  of  Communicating  Naturalistic 
Insights  for  System  Development  and  Evaluation  In  In  Proceedings  of  Naturalistic  Decision  Making  9.  London,  UK. 
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Key  Words: 
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Patterns,  socio-technical  systems,  system  evaluation,  pattern  language,  representation 

Summary  abstract: 

Motivation  -  The  Naturalistic  Decision  Making  (NDM)  community  benefits  from  having  its  insights  used  in 
software  development  projects. 

Research  approach  -  We  are  investigating  the  pattern  form  as  a  way  to  incorporate  findings  from  naturalistic 
research  into  software  development.  The  socio-technical  level  of  description  -  how  technology  impacts  groups, 
teams,  and  organizations  -  is  the  focus  of  our  work. 

Findings/Design  -  Pattern  form  offers  the  advantage  of  focusing  interpretation  and  connecting  to  a  solution  outline 
over  more  traditional  ways  of  making  findings  available.  These  advantages  also  make  the  pattern  form  good  for  use 
in  the  evaluation  of  systems  where  the  need  for  solutions  is  immediate. 

Research  limitations/Implications  -  The  pattern  form  is  not  a  panacea  and  will  not  replace  NDM  participation  in 
development  projects. 

Originality /Value  -  Patterns  have  not  been  used  for  evaluation  at  the  socio-technical  level. 

Take  away  message  -  Patterns  may  be  useful  for  communicating  NDM  insights  in  development  projects, 
particularly  those  at  the  socio-technical  level. 


Cox,  D.  A.,  &  Veinott,  E.  S.  (2010).  Ethno-Social  Framework  for  Evaluation  User’s  Guide. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Report 
Summary  abstract: 

The  focus  on  everyday  expertise  provided  by  the  European  sociological  ethnographers  of  work  practices,  such  as 
Fleath  and  Luff  (2000),  provides  a  valuable  complement  to  the  other  approaches  taken  (experimentation,  CWA, 
CTA)  in  the  Army  Research  Laboratory’s  Advanced  Decision  Architectures  program.  This  work  provides  detailed 
descriptions  of  situated  collaboration.  We  analyzed  selected  works  of  the  European  work  ethnographers  to  identify 
recurring  themes  about  how  collaborative  work  is  accomplished.  Our  aim  was  to  identify  these  themes  in  a  way  that 
would  be  useful  for  early  system  evaluation. 

We  developed  a  preliminary  tool  that  provides  the  initial  identification  of  the  kinds  of  alterations  that  may  occur  as  a 
result  of  using  a  collaborative  system  based  on  existing  ethnographic  research  and  methods.  This  provided  the  initial 
data  to  develop  a  method  for  early  reviews  of  collaborative  tools  in  order  to  anticipate  how  a  system  might  change 
individual  behavior,  team  dynamics,  and  organizational  workflow.  Although  this  is  a  preliminary  effort,  because 
collaborative  work  is  complex,  can  be  distributed,  and  occur  asynchronously,  we  feel  this  method  has  the  potential 
to  provide  useful  information  for  stakeholders  in  collaborative  programs  to  evaluate  the  systems  and  developers  of 
collaborative  system  to  reduce  misuses,  and  support  emergent  collaborative  processes  thereby  improving  transitions. 
We  identified  seven  themes:  coordinate  through  close  awareness  of  work  progress,  get  attention  while  minimizing 
disruption,  use  gesture  to  coordinate  conversations,  support  layers  of  context,  arrange  resources  and  practices  for 
efficient  communication,  coordinate  with  the  work  rather  than  explicitly  with  others,  and  negotiate  the  work  (Cox  & 
Veinott,  2009).  To  conduct  a  face  validity  check  on  the  themes  we  conducted  data  collection  on  the  tool  use  in  a 
highly  collaborative,  distributed,  and  sometimes  asynchronous  environment:  natural  disaster  response.  We 
interviewed  17  people  in  key  roles  for  responding  to  the  flooding  in  North  Dakota  that  occurred  in  Spring  2009.  We 
selected  this  incident  due  to  its  being  widely  viewed  as  an  example  of  effective  collaboration  in  emergency 
management  and  response.  We  conducted  interviews  with  the  North  Dakota  National  Guard,  civilian  emergency 
managers  (local  and  state),  and  volunteer  coordinators.  Our  interviews  focused  on  the  nature  of  their  collaborative 
work  and  the  role  of  technology  in  support  of  collaboration  (Veinott,  Cox,  &  Zimmerman,  in  press).  We  compared 
data  collected  on  Twitter™  during  the  2007  San  Diego  wildfires  (Cox  &  Veinott,  2009)  to  data  to  social  media  use 
during  the  North  Dakota  Floods. 

This  project  has  been  successful  in  extracting  from  the  work  of  the  European  sociological  ethnographers  themes  that 
can  be  applied  in  system  evaluation.  Future  research  will  extend  the  themes  to  cover  more  of  the  space  of  possible 
collaborative  work  arrangements,  validate  their  explanatory  and  predictive  power,  and  further  demonstrate  their 
utility  for  technical  audiences. 
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Crandall,  B.,  Klein,  G.,  &  Hoffman,  R.  R.  (2006).  Chapter  9.  Trends  and  Themes  in  the  Development  of  Cognitive 
Task  Analysis:  The  Rise  of  Modern  Cognitive  Psychology,  Working  Minds:  A  Practitioner's  Handbook  of 
Cognitive  Task  Analysis.  Cambridge,  MA:  MIT  Press. 
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Summary  abstract: 

Cognitive  Task  Analysis  as  a  set  of  methods  (and  a  community  of  practice)  began  to  emerge  in  the  early  1980s.  In 
this  chapter,  we  describe  the  historical  origins  and  more  recent  trends  and  influences  that  have  culminated  in  CTA. 
Cognitive  Task  Analysis  has  roots  in  a  number  of  disciplines.  More  than  just  a  historical  coincidence,  the 
simultaneous  emergence  of  CTA  in  a  number  of  areas  of  science  and  applied  research  is  a  reason  for  its  robustness. 
Understanding  those  multiple  traditions  provides  an  important  perspective  on  CTA  as  a  field  of  practice  (Woods, 
Tinapple,  Roesler,  and  Feil,  2002). 

Many  people  assume  that  CTA  emerged  as  a  consequence  of  the  rapid  evolution  of  computer-based  technology  and 
the  enormous  changes  in  the  workplace  that  have  occurred  in  parallel  In  fact,  CTA  is  deeply  rooted  in  the  history  of 
psychology. 

Beginning  with  the  writings  of  John  Watson  (1914),  the  paradigm  of  behaviorism  rose  to  prominence.  It  dominated 
American  academic  psychology  for  nearly  half  a  century,  despite  arguments  from  within  that  the  paradigm  had 
significant  limitations  (e.g.,  Lashley  1951).  But  in  the  1950s,  the  Carnegie  Institute  supported  interdisciplinary 
meetings  of  linguists,  psychologists,  and  computer  scientists  (Carroll  1953;  Cofer  1979;  Osgood  and  Sebeok  1954) 
that  marked  the  "psycholinguistic  revolution."  Among  the  champions  of  that  revolution  were  Noam  Chomsky,  who 
presented  notions  of  "generative  transformational  grammar"  and  criticisms  of  the  behaviorist  approach  to  language 
(Chomsky  1959). 

Notions  from  World  War  II-era  applied  research  also  had  an  impact  on  psychology.  Developments  in  audio¬ 
recording  and  signal-processing  technologies  led  to  significant  advances  in  research  on  speech  perception  (e.g., 
Liberman,  Delatre,  and  Cooper  1952).  Information  theory  and  signal  detection  theory  (e.g.,  Shannon  1948)  led  to 
research  on  perception,  attention,  and  vigilance. 


Crandall,  B.,  Klein,  G.,  &  Hoffman,  R.  R.  (2006).  Working  Minds:  A  Practitioner's  Handbook  of  Cognitive  Task 
Analysis.  Cambridge,  MA:  MIT  Press. 
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Summary  abstract: 

1  Introduction 

The  value  of  experience  and  of  understanding  how  to  apply  knowledge  is  described  in  a  wonderful  story,  a  story 
shared  so  often  it  has  become  one  of  those  enduring  urban  legends.  The  story  has  many  different  'versions’, 
involving  various  professions  and  famous  people.  In  one  version  of  the  story  the  scientist  and  inventor  Nikola  Tesla 
visited  Henry  Ford  at  his  automobile  factory.  The  factory  was  having  some  kind  of  difficulty  with  its  systems,  and 
Ford  asked  Tesla  if  he  could  help  identify  the  problem  area.  Tesla  \  .  walked  up  to  a  wall  of  boilerplates,  scanned 
them  briefly,  and  then  made  an  "X"  in  chalk  on  one  of  the  plates.  Examination  of  the  boilerplate  showed  that  it  was 
indeed  faulty.  Ford  was  impressed,  and  told  Tesla  to  send  an  invoice.  The  bill  arrived,  for  $10,000.  Ford,  never 
known  for  his  generosity,  was  astonished  at  the  cost  of  writing  an  "X"  on  the  boilerplate,  and  asked  for  a  breakdown. 
Tesla  sent  another  invoice,  which  read: 

Marking  wall:  $1 

Knowing  where  to  mark:  $9,999 
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This  story  speaks  directly  to  the  purposes  and  goals  of  this  book  in  two  respects.  First,  the  story  illustrates  the  "why" 
of  Cognitive  Task  Analysis  (CTA).  What  is  it  that  Tesla  knows,  and  how  does  he  know  it?  What  tells  him  what  to 
do,  with  Henry  Ford  (not  the  most  patient  of  men,  by  many  accounts)  looking  over  his  shoulder?  Capturing  that 
knowledge  and  reasoning  is  one  of  the  things  CTA  can  do. 

Second,  the  story  illustrates  the  "how"  of  CTA.  Cognitive  Task  Analysis  can  be  thought  of  as  a  set  of  tools  in  a 
toolkit.  Like  any  tool,  CTA  can  be  employed  well  and  wisely,  or  it  can  be  employed  poorly  or  inappropriately.  What 
tool  would  you  use  if  you  wanted  to  understand  how  Tesla  was  able  to  grasp  the  nature  of  the  problem  so  quickly? 
This  book  is  about  having  the  tools  and  the  toolkit  to  understand  how  people  think:  how  their  minds  work,  what  they 
struggle  with,  and  how  they  manage  to  perform.  The  book  is  organized  into  three  major  sections: 

Part  I,  "Tools  for  Exploring  Cognition  in  Context,"  provides  detailed  guidance  for'  planning  and  carrying  out  CTA. 

It  includes  chapters  on  capturing  knowledge  and  on  capturing  the  way  people  reason.  We  rely  on  this  distinction 
throughout  the  book:  CTA  investigates  what  people  know  and  how  they  think. 

Part  II,  "Finding  Cognition,"  provides  a  perspective  on  studying  cognition  in  real-world  settings  and  what  an 
expanded  view  of  cognition-a  macrocognitive  framework-offers.  We  describe  some  of  the  issues  that  surround  CTA 
and  what  it  means  to  study  cognition  in  context.  We  end  the  section  by  exploring  the  challenges  of  rapidly  changing 
technology. 

Part  III,  "Putting  CTA  Findings  to  Use,"  describes  key  issues  in  applying  CTA  findings  to  several  applications  areas: 
technology  development,  .-raining  and  instructional  design,  and  market  research.  We  also  present  a  chapter  on  the 
role  of  CTA  in  the  development  of  measures  for  evaluating  cognitive  work. 

Our  intent  in  writing  this  book  is  to  share  what  we  have  learned  about  CTA,  from  our  experience  in  the  field  to  the 
concepts  and  models  we  draw  on.  We  have  offered  examples  and  suggested  ways  to  apply  CT  A  findings  to  real- 
world  problems  and  issues.  We  hope  this  book  provides  you  with  some  tools  you  can  use  in  your  own  practice  and 
that  the  CT  A  methods  can  help  you  discover  how  people  like  Tesla  know  where  to  put  their  chalk  marks. 

2  Overview  of  Cognitive  Task  Analysis  Methods 

We  have  reviewed  many  different  approaches  to  CTA  in  order  to  show  the  possibilities  that  exist  and  to  provide  a 
context  for  the  methods  we  describe  in  part  I.  Another  reason  for  this  overview  is  to  demonstrate  that  there  is  no 
single  right  way  to  do  CTA.  Practitioners  of  CTA  have  a  wide  range  of  choices  in  the  strategy  to  use  in  knowledge 
elicitation,  data  analysis,  and  knowledge  representation.  Instead  of  worrying  about  following  an  official  program, 
practitioners  are  better  served  by  tracking  the  cognitive  phenomenon  they  want  to  understand.  Getting  an  insightful 
account  of  this  phenomenon  is  far  more  important  than  preserving  methodological  rigor  that  might  interfere  with  the 
investigation. 

Cognitive  Task  Analysis  research  is  often  conducted  as  field  studies,  since  it  com-  prises  the  initial  exploration  of  a 
cognitive  process  or  strategy  that  is  not  well  understood.  We  argue  that  it  is  misplaced  rigor-rigor  mortis,  in  fact-to 
let  the  choice  of  methods  overshadow  the  phenomenon  being  studied.  For  field  research,  scientific  values  dictate 
that  the  methods  you  employ  be  documented  and  that  your  analyses  be  described  in  sufficient  detail  so  that  others 
can  review  your  efforts  and  replicate  your  findings.  You  will  also  want  to  document  evidence  that  runs  counter  to 
your  hypotheses.  These  are  all  appropriate  measures  to  increase  scientific  validity.  In  contrast,  a  rigid  adherence  to 
experimental  control  during  a  CTA  study  is  an  inappropriate  attempt  to  mimic  the  psychology  laboratory. 

Therefore,  we  recommend  that  CTA  researchers  be  prepared  with  a  range  of  methods  that  they  can  use  or  adapt. 
Researchers  have  many  choices  in  the  strategy  they  use  in  knowledge  elicitation,  data  analysis,  and  knowledge 
representation.  The  remainder  of  this  book,  in  a  sense,  is  aimed  at  providing  enough  information  .  so  that  researchers 
can  make  those  choices. 

I.  Tools  for  Exploring  Cognition  in  Context 

3  Preparation  and  Framing 

This  chapter  described  activities  that  provide  the  initial  preparation  for  a  successful  CTA  project.  A  set  of  framing 
questions  were  presented  that  can  provide  important  insights  and  guidance  throughout  the  entire  project.  Answers  to 
the  framing  questions  will  result  in  some  ideas  and  initial  decisions  about: 

•  the  issue  or  need  that  the  research  will  address, 

•  the  kind  of  deliverable  or  product  that  will  result, 

•  the  kinds  of  individuals  who  might  be  best  as  "target  users"  or  "informantcollaborators," 

•  the  aspects  of  cognition  or  expertise  that  must  be  revealed, 

•  the  kinds  of  research  settings  that  might  be  most  appropriate  to  the  project  goals,  and 

•  the  kinds  of  individuals  who  would  be  the  best  participants  in  the  CTA  procedures. 
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The  chapter  also  provided  guidance  about  how  to  work  with  task-function  descriptions  that  may  already  exist  and 
ways  to  come  up  to  speed  and  learn  about  the  domain. 

4  Using  Concept  Maps  for  Knowledge  Elicitation  and  Representation 

CTA  involves  capturing  what  practitioners  know  about  their  domain:  its  concepts,  principles,  and  events.  We  can 
think  of  no  CTA  process  or  project  in  which  the  CTA  researchers  did  not  have  to  elicit  and  then  represent  at  least 
some  domain  knowledge.  This  chapter  reviews  the  procedures  and  applications  of  Concept  Mapping  as  a  proven 
knowledge  elicitation  method  for  the  efficient  elicitation  of  practitioner  knowledge.  Concept  Maps  involve  labeled 
nodes  and  links,  but  Concept  Maps  differ  in  important  ways  from  other  types  of  diagrams  that  utilize  combinations 
of  graphical  and  textual  elements  to  represent  or  express  meanings.  Concept  Mapping  supports  the  practitioner's 
effort  to  reach  for  crystal  clarity  about  what  he  or  she  wishes  to  express.  In  Concept-Mapping  knowledge  elicitation, 
the  researchers  help  the  domain  practitioner  build  up  a  representation  of  domain  knowledge,  in  effect  merging  the 
activity  of  knowledge  elicitation  and  the  activity  of  knowledge  representation. 

5  Incident-Based  CTA:  Helping  Practitioners  "Tell  Stories" 

In  this  chapter,  we  described  one  method  for  using  incidents  to  extract  cognitive  elements-the  Critical  Decision 
Method.  We  described  the  procedures  for  conducting  a  CDM  interview  and  offered  an  interviewer's  perspective  on 
each  of  the  CDM  components.  We  examined  the  boundary  conditions  under  which  CDM  is  less  likely  to  be 
effective,  and  we  described  some  of  the  variations  and  adaptations  that  have  developed  to  take  advantage  of  the  data 
collection  opportunities  that  real,  lived  experience  offers. 

6  CTA  Methods  and  Experiment-Like  Tasks 

In  this  chapter  we  explored  some  intersections  of  laboratory  methods  and  field  research  methods  for  conducting 
CTA.  Researchers  can  tinker  with  aspects  of  a  familiar  task  in  a  variety  of  ways,  thereby  eliciting  experts'  strategies 
and  reasoning,  and  do  so  in  structured  ways  that  provide  empirical  leverage.  We  also  discussed  think-aloud 
problem-solving  and  protocol  methods  and  the  use  of  analytic  and  representational  formats  that  can  reveal  important 
aspects  of  cognitive  process. 

7  Analysis  and  Representation 

The  intent  in  this  chapter  has  been  to  open  the  "black  box"  of  analysis  and  representation,  and  examine  the  processes 
and  procedures  that  can  be  used  to  transform  specific  data  elements  to  general  findings.  Reading  about  CTA  is  likely 
to  lead  to  an  impression  that  knowledge  elicitation  activities  constitute  the  entire  CTA  process  and  that  once  the  data 
have  been  gathered  one  can  move  easily,  with  little  time  or  effort,  into  application  of  the  findings.  For  a  number  of 
CTA  techniques,  the  knowledge  elicitation  methods  do  produce  a  data  structure  and  populate  it.  In  some  methods, 
such  as  concept  mapping  or  the  simulation  interview  in  ACTA,  knowledge  elicitation  and  knowledge  representation 
are  inherently  linked.  But  these  techniques  still  produce  representations  on  an  individual  level,  specific  to  a 
particular  SME,  simulation,  or  event.  They  do  not  circumvent  the  necessary  steps  of  integrating  and  synthesizing 
data,  extracting  meaning,  and  identifying  key  findings.  How  those  activities  are  carried  out,  and  how  findings  and 
applications  link  back  to  individual  data  elements,  requires  consistent  attention  and  clear  explanation. 

II.  Finding  Cognition 

8  Thinking  About  Cognition 

In  this  chapter  we  explored  what  it  means  to  study  cognition  in  natural  contexts  and  how  that  differs  from  the  study 
of  cognition  in  a  laboratory.  We  described  a  cognitive  landscape  that  includes  aspects  of  expertise.  This  perspective 
helps  to  uncover  subtle  aspects  of  cognition  in  real-world  contexts.  The  cognitive  landscape  includes 
macrocognition-the  cognitive  functions  and  processes  that  emerge  when  the  research  focus  shifts  to  natural  contexts. 
We  discussed  how  field  studies  of  cognition  and  appropriate  use  of  qualitative  methodologies  meet  the  criteria  for 
scientific  inquiry.  These  topics  matter  because  there  is  more  to  CTA  practice  than  knowledge  of  CTA  tools  and 
techniques.  The  best  CTA  practitioners  spend  time  learning  about  cognitive  elements  of  performance  in  order  to 
understand  a  wide  range  of  cognitive  phenomena.  By  understanding  the  way  people  think  and  reason  in  natural 
contexts,  CTA  practitioners  are  more  likely  to  recognize  important  aspects  of  cognition  when  they  encounter  them. 

9  Trends  and  Themes  in  the  Development  of  Cognitive  Task  Analysis:  The  Rise  of  Modern  Cognitive  Psychology 
In  this  chapter  we  explored  the  historical  roots  of  CTA.  Academic  research  traditions  and  trends  in  applied  research 
have  each  contributed  to  the  development  of  CTA  methods.  Reaction  against  the  dominance  of  behaviorism  in  North 
American  psychology  led  researchers  to  a  renewed  interest  in  understanding  cognitive  processes.  This  was 
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accelerated  by  the  advent  of  information  theory  and  computer  metaphors  of  mind.  In  parallel,  the  impact  of 
technology  in  the  workplace  prompted  applicationoriented  researchers  to  explore  a  wider  array  of  complex  social- 
technological  influences.  The  confluence  of  factors  produced  a  wave  of  interest  in  cognitive  work  and  led 
researchers  in  a  number  of  communities  of  practice  to  discuss  what  have  come  to  be  called  CTA  methods. 

10  Information  Technology 

In  designing  and  introducing  new  types  of  IT,  there  are  many  decisions  to  be  made  about  how  people  will  work  with 
the  technology-how  they  will  coordinate  with  it,  what  roles  the  technology  will  play,  and  how  to  allocate  functions 
between  people  and  technology.  In  this  chapter  we  described  some  of  the  challenges  and  opportunities  that  arise 
when  IT  is  introduced  to  help  people  perform  tasks  and  some  of  the  cognitive  demands  that  are  created  by  IT. 

III.  Putting  CTA  Findings  to  Use 

1 1  The  Role  of  Cognitive  Requirements  in  System  Development 

In  this  chapter,  we  examined  some  specific  ways  to  use  CTA  to  generate  cognitive  requirements  for  system 
development  and  design.  We  presented  a  particular  approach,  DCD,  in  order  to  illustrate  and  describe  some  general 
principals  of  cognitive  engineering.  Information  technology  offers  enormous  potential  for  improving  decision 
making,  but  only  if  the  developers  of  technologies  take  cognitive  requirements  into  account,  not  just  at  the  test  and 
evaluation  stage,  but  throughout  the  development  and  procurement  process. 

12  Cognitive  Training 

CTA  is  central  to  cognitive  training.  Uyou  want  to  improve  cognitive  functions  such  as  decision  making, 
sensemaking,  problem  detection,  replanning,  and  so  forth,  you  will  need  to  understand  how  these  functions  are 
accomplished  and  where  people  struggle  and  run  into  difficulties.  In  this  way,  you  can  define  cognitive  training 
requirements. 

The  primary  applications  of  CTA  for  cognitive  training  are  to  identify  and  define  the  training  requirements,  discover 
the  basis  of  skills,  the  source  of  difficulties,  collect  incidents  that  can  be  transformed  into  training  scenarios,  and 
guide  the  feedback  process  to  help  trainees  learn  from  their  experiences. 

One  question  we  are  often  asked  is  about  the  relationship  between  cognitive  skills  training  based  on  CTA  and  more 
traditional  training  approaches  such  as  instructional  systems  design  (1SD).  ISD  typically  has  been  used  for 
procedural  tasks;  so  cognitive  skills  training  could  complement  ISD  by  providing  a  cognitive  perspective. 

However,  specifying  the  cognitive  functions  has  the  potential  to  make  ISD  more  cumbersome  and  harder  to  use. 
Critics  of  ISD  (e.g.,  Gordon  and  Zemke  2000)  see  the  ISD  approach  as  already  being  unwieldy,  and  inefficiently 
carving  the  world  into  too  many  small  slices.  Adding  the  cognitive  dimension  could  make  ISD  even  more  so  by 
overlaying  all  the  procedural  objectives  with  cognitive  ones. 

CTA  efforts  should  result  in  richer  training  objectives,  not  longer  lists  of  objectives.  We  do  not  see  the  decision 
requirements  as  a  discrete  set  of  training  objectives.  Rather,  they  are  different  facets  of  performing  the  cognitive 
work.  Training  developers  would  not  want  separate  modules,  say,  for  problem  detection  skills  and  sensemaking 
skills.*  The  same  scenarios  should  be  training  sensemaking  and  problem  detection  and  decision  making.  The  design 
process  in  cognitive  training  aims  at  providing  experiences  that  permit  trainees  to  explore,  reflect,  learn,  work 
through  confusion,  and  develop  deeper  and  richer  mental  models  for  carrying  out  complex  tasks. 

13  Understanding  How  Consumers  Make  Decisions:  Using  Cognitive  Task  Analysis  for  Market  Research 
Market  researchers  are  primarily  interested  in  figuring  out  how  to  influence  consumer  behavior.  Conventional 
approaches  to  market  research  have  pursued  that  goal  by  studying  consumer  behavior,  attitudes,  deSires,  and  beliefs. 
Although  all  this  information  is  valuable  and  important,  it  does  not  address  the  critical  component  of  consumer 
cognition:  what  consumers  know,  how  they  think,  and  what  strategies  they  have  developed  for  buying  and  using 
products.  This  chapter  explored  ways  in  which  CTA  methods  can  be  applied  to  market  research  questions.  Cognitive 
Task  Analysis  methods  can  reveal  the  strategies  consumers  use  to  make  the  purchase  decision-whether  to  purchase  a 
particular  product,  which  brand  to  select,  which  features  to  choose,  or  whether  to  try  a  different  brand  with  different 
features  altogether.  Cognitive  Task  Analysis  methods  can  explain  how  users  understand  what  products  do  and  how 
they  work,  and  can  offer  insights  about  why  consumers  may  not  use  products  in  the  ways  that  their  developers 
intend.  Finally,  CTA  methods  can  reveal  the  skills  and  the  gaps  in  understanding  that  front-line  sales  staff  may  have 
regarding  their  customers. 

14  Cognitive  Task  Analysis  for  Measurement  and  Evaluation 
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Measurement  of  various  kinds  and  combinations  is  critical  in  shaping  CTA,  conducting  CTA,  and  going  from  CTA 
results  to  conclusions  and  recommendations.  Cognitive  Task  Analysis  methods  are  well  suited  to  identify  cognitive 
requirements  and  to  support  the  creation  and  application  of  cognitive  measures  to  study  macrocognitive  functions. 
With  the  growing  use  of  information  technologies,  developers  need  to  determine  how  their  systems  are  affecting 
cognitive  processes.  Cognitive  measures  are  relevant  to  software  support  systems.  Cognitive  measures  are  also 
needed  with  the  variety  'of  training  programs  aimed  at  improving  decision  making  and  other  macrocognitive 
functions.  We  invite  all  of  these  communities  to  borrow  and  apply  CTA  probes  and  methods  as  they  find  practical. 
Practitioners  may  want  to  conduct  CTA  studies,  or  they  may  just  need  to  tap  into  cognitive  requirements  in  a 
piecemeal  fashion.  Whichever  is  the  case,  deploying  CTA  methods  in  the  evaluation  process  should  help  produce 
results  that  lead  to  better  system  performance. 

15  Future  Directions  for  Cognitive  Task  Analysis 

We  hope  that  throughout  the  reading  of  this  book  you  have  learned  some  useful  methods  along  with  some  effective 
ways  of  implementing  these  methods.  We  have  tried  to  describe  our  trade  secrets,  to  help  you  understand  various 
CTA  methods  and  to  give  you  the  courage  to  explore  new  approaches  to  gathering  data.  We  have  tried  to  show  the 
depth,  richness,  and  value  of  CTA  and  its  applications.  As  you  can  see,  there  are  no  hidden  mysteries,  only  hard 
work  and  practice,  as  with  mastering  all  complex  skills. 

Appendix 
-  The  Interview 

•  Establish  rapport  with  the  participant 

•  Know  how  to  use  time  to  greatest  impact-when  to  spend  time  on  a  topic  and  when  to  move  on 

•  Ask  good  opening  questions:  they  know  what  they  want  and  how  to  get  it  efficiently 

•  Recognize  where  to  drill  down,  what  to  deepen  on;  they  can  hear  words  or  phrases  that  flag  rich  areas  for  probing 

•  Recognize  when  a  direction  is  not  fruitful:  they  can  turn  the  interview  around  and  get  it  going  in  a  productive 
direction 

•  Recognize  how  to  frame  questions:  how  to  pose  a  question  so  that  it  makes  sense  but  doesn't  lead  the  interviewee 

•  Recognize  where  the  important  content  is  likely  to  be  and  how  to  bring  the  interview  to  that  place  (this  may  require 
a  whole  series  of  questions  that  set  up  a  key  insight) 

•  Know  how  to  reorient  the  interview  when  the  current  direction  isn't  working 

•  Have  a  range  of  strategies  for  probing,  deepening  into  content 

•  Understand  the  power  of  silence:  they  can  wait  for  the  interviewee  to  think  about  a  question  without  filling  the 
void  with  talk 
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Summary  abstract: 

There  is  a  growing  reliance  on  automation  (e.g.,  intelligent  agents,  semi-autonomous  robotic  systems)  to  effectively 
execute  increasingly  cognitively  complex  tasks.  Successful  team  performance  for  such  tasks  has  become  even  more 
dependent  on  team  cognition,  addressing  both  human-human  and  human-automation  teams.  Team  cognition  can  be 
viewed  as  the  binding  mechanism  that  produces  coordinated  behavior  within  experienced  teams,  emerging  from  the 
interplay  between  each  team  member’s  individual  cognition  and  team  process  behaviors  (e.g.,  coordination, 
communication).  In  order  to  better  understand  team  cognition  in  human-automation  teams,  team  performance 
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models  need  to  address  issues  surrounding  the  effect  of  human-agent  and  human-robot  interaction  on  critical  team 
processes  such  as  coordination  and  communication.  Toward  this  end,  we  present  a  preliminary  theoretical 
framework  illustrating  how  the  design  and  implementation  of  automation  technology  may  influence  team  cognition 
and  team  coordination  in  complex  operational  environments.  Integrating  constructs  from  organizational  and 
cognitive  science,  our  proposed  framework  outlines  how  information  exchange  and  updating  between  humans  and 
automation  technology  may  affect  lower-level  (e.g.,  working  memory)  and  higherlevel  (e.g.,  sense  making) 
cognitive  processes  as  well  as  teams’  higherorder  “metacognitive”  processes  (e.g.,  performance  monitoring).  Issues 
surrounding  human-automation  interaction  are  discussed  and  implications  are  presented  within  the  context  of 
designing  automation  technology  to  improve  task  performance  in  human-automation  teams. 
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Summary  abstract: 

Technological  advances  have  significantly  intensified  the  challenges  facing  teams  performing  in  today’s  operational 
environments  (e.g.,  nuclear  power  plants,  military  battlefield,  long  duration  space  flight),  leading  to  a  growing 
reliance  on  automation  (e.g.,  intelligent  agents  and  semi-autonomous  robotic  systems)  to  support  and  enhance  team 
performance  in  increasingly  complex  cognitive  tasks.  Integrating  constructs  from  organizational  and  cognitive 
science,  we  describe  a  theoretical  framework  illustrating  how  the  design  and  implementation  of  automation 
technology  may  influence  team  cognition  and  decision  making.  We  also  discuss  the  results  of  a  preliminary  study, 
conducted  in  a  military  command  and  control  environment,  aimed  at  exploring  components  of  our  theoretical 
framework,  specifically  focusing  on  the  attitudinal  variables  (human  operators’  attitudes  toward  automation)  that 
may  influence  human-automation  interaction. 


Davis,  J.  W.,  Morison,  A.,  &  Woods,  D.  D.  (2007).  Building  Adaptive  Camera  Models  for  Video  Surveillance.  In  In 
Proceedings  of  IEEE  Workshop  on  Applications  of  Computer  Vision.  Austin,  TX. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings;  Video 
Surveillance 

Summary  abstract: 

We  address  the  limited  automatic  scanning  functionality  of  standard  PTZ  camera  systems.  We  present  an  adaptive, 
scene-specific  model  using  standard  PTZ  camera  hardware.  The  adaptive  model  is  constructed  automatically  by 
detecting  human  activity  in  Motion  History  Images  (MHIs)  using  an  iterative  candidacy-classification-reduction 
process.  The  target  motion  is  quantified  and  employed  in  the  construction  of  a  global  Activity  Map,  which  in  turn  is 
used  to  direct  or  navigate  the  camera. 


Davis,  J.  W.,  Morison,  A.  M.,  &  Woods,  D.  D.  (2007).  An  Adaptive  Focus-of-Attention  Model  for  Video 
Surveillance  and  Monitoring.  Machine  Vision  and  Applications  Journal,  18(1),  41-64. 

Key  Words: 
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Journal  Article 

Computer  vision  •  Machine  vision  • 

Motion  detection  •  Surveillance  ■  Security  •  Camera 
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scanning  •  Motion  history  images 
Summary  abstract: 

In  current  video  surveillance  systems,  commercial  pan/tilt/zoom  (PTZ)  cameras  typically  provide  naive  (or  no) 
automatic  scanning  functionality  to  move  a  camera  across  its  complete  viewable  field.  However,  the  lack  of  scene- 
specific  information  inherently  handicaps  these  scanning  algorithms.  We  address  this  issue  by  automatically 
building  an  adaptive,  focusof-  attention,  scene-specific  model  using  standard  PTZ  camera  hardware.  The  adaptive 
model  is  constructed  by  first  detecting  local  human  activity  (i.e.,  any  translating  object  with  a  specific  temporal 
signature)  at  discrete  locations  across  a  PTZ  camera’s  entire  viewable  field.  The  temporal  signature  of  translating 
objects  is  extracted  using  motion  history  images  (MHIs)  and  an  original,  efficient  algorithm  based  on  an  iterative 
candidacy-  classification-reduction  process  to  separate  the  target  motion  from  noise.  The  target  motion  at  each 
location  is  then  quantified  and  employed  in  the  construction  of  a  global  activity  map  for  the  camera.  We  additionally 
present  four  new  camera  scanning  algorithms  which  exploit  this  activity  map  to  maximize  a  PTZ  camera’s 
opportunity  of  observing  human  activity  within  the  camera’s  overall  field  of  view.  We  expect  that  these  efficient  and 
effective  algorithms  are  implementable  within  current  commercial  camera  systems. 


Dekker,  S.  W.  A.,  Nyce,  J.  M.,  &  Hoffman,  R.  R.  (2003).  From  Contextual  Inquiry  to  Designable  Futures:  What  Do 
We  Need  to  Get  There?  IEEE  Intelligent  Systems,  18(2),  74077. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

So,  what  is  an  end  user,  really?  And  what  do  we  really  mean  by  the  procurement  process?  Human  factors 
researchers  often  take  certain  agendas,  terms,  and  theories  for  granted  or  rely  on  them  out  of  habit.  This  essay  takes 
a  special  look  at  contextual  enquiry  as  a  putatively  (and  indeed  potentially)  superior  way  of  giving  end  users  a 
serious  say  in  the  procurement  process  of  complex  cognitive  systems. 


Dekker,  S.  W.  A.,  &  Woods,  D.  D.  (2003).  Maba  Maba  or  Abracadabra?  Progress  in  Human-Automation 
Coordination.  Cognition,  Technology,  and  Work,  44(4),  240-244. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Automation;  Cognition;  Computers;  Co-ordination;  Function  allocation;  Human  error 
Summary  abstract: 

In  this  paper  we  argue  that  substitution-based  function  allocation  methods  (such  as  MABA-MABA,  or  Men-Are- 
Better-At/Machines-Are-  Better-At  lists)  cannot  provide  progress  on  human-automation  co-ordination.  Quantitative 
‘who  does  what’  allocation  does  not  work  because  the  real  effects  of  automation  are  qualitative:  it  transforms  human 
practice  and  forces  people  to  adapt  their  skills  and  routines.  Rather  than  re-inventing  or  refining  substitution-based 
methods,  we  propose  that  the  more  pressing  question  on  human-automation  co-ordination  is  ‘How  do  we  make 
them  get  along  together?’ 


dePontbriand,  R.  J.,  Woods,  D.,  Schultz,  A.  C.,  Archer,  S.,  Endsley,  M.,  &  Scholtz,  J.  (2003).  Discussion  Panel: 
Human  Robot  Coordination.  Intent  at  a  Distance;  Dynamic  Level  of  Autonomy;  Soldier  Performance  Assessment 
for  Control  of  Multiple  Unmanned  Systems;  Task  Interruption  When  Interacting  with  Robotic  Systems;  Supporting 
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Situation  Awareness  in  Humanrobot  Collaboration;  Evaluation  of  Human-Robot  Collaboration.  In  In  Proceedings  of 
47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  recent  years,  accelerating  maturation  of  robotics  technologies,  such  as  machine  perception  and  intelligent  control 
technologies  have  led  to  a  widening  knowledge  and  experiential  base  serving  as  a  springboard  for  expanded 
research. 

This  maturation  has  led  to  increased  engineering  reliability,  opening  the  way  for  more  tractable  human-robot 
research.  The  technology  area  from  which  a  good  deal  of  this  work  can  draw  inspiration  and  guidance  is  humans  in 
automation  (Parasuraman,  Sheridan  and  Wickens,  2000),  which  lets  us  examine  issues  including  air  traffic 
controllers’  multitracking  or  pilots’  multi-tasking  operations  under  conditions  of  time  and  other  stressors.  Among 
the  issues  the  panelists  will  address  are:  How  will  human-robot  teams  dynamically  reconfigure  or  gracefully  degrade 
as  assets  are  lost?  How  can  people  and  robots  make  judgments  of  the  robots  ability  to  traverse  or  climb  broken 
terrain?  How  can  people  manage  multiple  robots?  How  can  robots  and  people  build  a  shared  awareness  of  the 
remote  environment?  What  metrics  will  capture  human-robot  teamwork?  Overall,  human-robot  team  work  is  a  new 
frontier  for  Human  Factors. 


Diesner,  J.,  &  Carley,  K.  (2009).  Looking  under  the  Hood  of  Stochastic  Machine  Learning  Algorthms  for  Parts  of 
Speech  Tagging.  In  In  Proceedings  of  Conference  at  Carnegie  Mellon  University,  School  of  Computer  Science, 
Institute  for  Software  Research,  Technical  Report  CMU-ISR-08-13 1R. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Part  of  Speech  Tagging,  Hidden  Markov  Models,  Viterbi  Algorithm,  AutoMap 

Summary  abstract: 

A  variety  of  Natural  Language  Processing  and  Information  Extraction  tasks,  such  as  question  answering  and  named 
entity  recognition,  can  benefit  from  precise  knowledge  about  a  words"  syntactic  category  or  Part  of  Speech  (POS) 
(Stolz,  Tannenbaum  et  al.  1965;  Church  1988;  Rabiner  1989).  POS  taggers  are  widely  used  to  assign  a  single  best 
POS  to  every  word  in  text  data,  with  stochastic  approaches  achieving  accuracy  rates  of  up  to  96  to  97  percent 
(Jurafsky  and  Martin  2000).  When  building  a  POS  tagger,  human  beings  needs  to  make  a  set  of  decisions,  some  of 
which  significantly  impact  the  accuracy  and  other  performance  aspects  of  the  resulting  engine.  In  this  paper  we 
provide  an  overview  of  these  decisions  and  empirically  determine  their  impact  on  POS  tagging  accuracy.  We 
envision  the  gained  insights  to  be  a  valuable  contribution  for  people  who  want  to  design,  implement,  modify,  fine- 
tune,  integrate,  or  simple  reasonably  use  a  POS  tagger.  Based  on  the  results  presented  herein  we  built  and  integrated 
a  POS  tagger  into  AutoMap,  a  tool  that  facilitates  Natural  Language  Processing  and  relational  text  analysis,  as  a 
stand-alone  feature  as  well  as  an  auxiliary  for  other  tasks. 


Diesner,  J.,  &  Carley,  K.  M.  (2008).  Conditional  Random  Fields  for  Entity  Extraction  and  Ontological  Text  Coding. 
Journal  of  Computational  and  Mathematical  Organization  Theory,  13,  248-262. 

Key  Words: 
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Advanced  Decision  Architectures 
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Ontological  Text  Coding  •  Semantic  networks  •  Entity  Extraction  • 

Supervised  machine  learning  •  Conditional  models  •  Conditional  Random  Fields 

Summary  abstract: 
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Previous  research  suggests  that  one  field  with  a  strong  yet  unsatisfied  need  for  automatically  extracting  instances  of 
various  entity  classes  from  texts  is  the  analysis  of  socio-technical  systems  (Feldstein  in  Media  in  Transition  MiT5, 
2007;  Hampe  et  al.  in  Netzwerkanalyse  und  Netzwerktheorie,  2007;  Weil  et  al.  in  Proceedings  of  the  2006 
Command  and  Control  Research  and  Technology  Symposium,  2006;  Diesner  and  Carley  in  XXV  Sunbelt  Social 
Network  Conference,  2005).  Traditional  as  well  as  non-traditional  and  customized  sets  of  entity  classes  and  the 
relationships  between  them  are  often  specified  in  ontologies  or  taxonomies.  We  present  a  Conditional  Random  Fields 
(CRF)-based  approach  to  distilling  a  set  of  entities  that  are  defined  in  an  ontology  originating  from  organization 
science.  CRF,  a  supervised  sequential  machine  learning  technique,  facilitates  the  derivation  of  relational  data  from 
corpora  by  locating  and  classifying  instances  of  various  entity  classes.  The  classified  entities  can  be  used  as  nodes 
for  the  construction  of  socio-technical  networks.  We  find  the  outcome  sufficiently  accurate  (82.7  percent  accuracy 
of  locating  and  classifying  entities)  for  future  application  in  the  described  problem  domain.  We  propose  using  the 
presented  methodology  as  a  crucial  step  in  the  process  of  advanced  modeling  and  analysis  of  complex  and  dynamic 
networks. 


Diggelen,  J.  v.  (2009).  Fulfilling  Collective  Obligations  in  Fluman-Agent  Teams  Using  KAoS  Policies.  In  In 
Proceedings  of  Workshop  on  Rich  Cognitive  Models  for  Policy  Design  and  Simulation,  Lorentz  Center,  Leiden,  The 
Netherlands. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

Autonomous  agents 

•  A  key  property  of  agents  is  autonomy,  meaning  that  they  can  operate  on  their  own  without  the  need  for  external 
guidance 

•  Research  on  autonomy  has  focused  on  endowing  an  agent  with  control  over  its  own  actions  and  internal  state 

•  Systems  that  consist  of  autonomous  agents  are  particularly  suited  to  implement  open  systems,  heterogeneous 
systems,  complex  systems 


Dominguez,  C.,  Cox,  D.,  William  G.  Long,  Moon,  B.,  &  Klein,  G.  (2005).  Helping  Analysts  Deal  with  Data 
Overload:  Profiling  Profilers.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Intelligence  analysts  provide  an  incredibly  important  service  toward  maintaining  our  national  security.  They  are 
chartered  with  providing  the  knowledge  basis  for  decisions  our  national  and  military  leaders  make.  Analysts  must  be 
depended  upon  to  influence  high-risk,  time-critical,  sometimes  life-or-death  decisions;  thus,  supporting  their 
requirements  is  a  vital  goal  for  research  and  development  efforts.  In  this  paper,  we  focus  on  the  cognitive  aspects  of 
analysts’  information  collection  and  filtering  tasks,  particularly  of  all-source  analysts  who  continuously  monitor 
message  traffic.  These  analysts  create  synthesized  reports  for  customers.  We  will  discuss  the  issue  of  data  overload 
and  its  effects,  along  with  current  lines  of  research  on  information  retrieval.  We  end  with  a  summary  of  our  research 
effort  examining  filtering/profiling,  to  include  approach  and  initial  findings,  towards  our  eventual  goal  of  developing 
design  seeds  that  can  be  incorporated  into  software  tools  to  better  support  analysts’  cognitive  work. 


Dominguez,  C„  Uhlig,  P.,  Brown,  J.,  Gurevich,  O.,  Shumar,  W.,  Stahl,  G.,  Zemel,  A.,  &  Zipperer,  L.  (2005). 
Studying  and  Supporting  Collaborative  Care  Processes.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society.  Orlando,  FL. 
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Summary  abstract: 

In  patient  care  today,  teams  of  practitioners  from  various  disciplines  must  coordinate  their  efforts  in  order  to  deliver 
care  successfully.  Frontline  nurses  and  physicians  must  interact  with  social  workers,  therapists,  physician  assistants, 
nurse  practitioners,  and  others  to  develop  and  carry  out  coordinated  plans  of  care.  Also,  clinical  team  members  must 
communicate  with  patients  and  their  families  in  language  that  can  be  understood  and  acted  upon.  In  support  of  these 
goals,  JCAHO  standards  require  patient  care  to  be  planned  and  provided  in  an  interdisciplinary,  collaborative 
manner.  As  hospital  units  develop  processes  for  collaborative  care  in  complex  environments  such  as  post-surgery 
and  critical  care  units,  it  is  important  to  understand  what  constitutes  success  for  these  processes  and  how  they  can  be 
enabled  and  supported.  This  report  documents  a  series  of  field  visits  and  simulations  designed  to  observe,  videotape, 
and  interview  collaborative  care  team  members,  patients,  and  family  members  engaged  in  varying  forms  of 
collaborative  practice.  This  ongoing  research  is  being  conducted  by  a  multi-disciplinary  team  of  medical  and  social 
scientists  with  a  shared  goal  of  studying  and  supporting  collaborative  care  processes. 


Donnelly,  J.,  Madden,  J.,  Roberts,  A.,  Greenberg,  M.,  Bradshaw,  J.,  &  Uszok,  A.  (2007).  Dynamic  Policy 
Enforcement  in  Jbi  Information  Services  with  the  KAoS  Policy  and  Domain  Services.  In  In  Proceedings  of  SPIE 
Defense  and  Security  Symposium,  Orlando,  FL. 


Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
dynamic  policy  enforcement,  information  management,  JBI,  KAoS,  ontology,  OWL 

Summary  abstract: 

English-language  policies  about  the  desired  behavior  of  computer  systems  often  suffer  from  translation  errors  when 
implemented  with  a  proliferation  of  low-level  rules  governing  access  control,  resource  allocation  and  configuration. 
To  solve  this.  Dynamic  Policy  Enforcement  systems  replace  these  low-level  rules  with  a  relatively  small  number  of 
semantically  grounded,  machine-understandable  policy  statements.  These  statements  use  domain  terms  defined  in  an 
ontology  that  are  formally  defined  so  that  they  can  be  enforced  by  the  system  but  also  meaningful  to  human 
administrators  to  ensure  that  they  accurately  represent  organizational  policies.  In  this  paper,  we  describe  the 
application  of  one  such  Dynamic  Policy  Enforcement  system,  KAoS,  to  the  control  of  distributed,  information- 
management  services  defined  by  the  Air  Force  Research  Laboratory's  Joint  Battlespace  Infosphere  (JBI)  program. 
Our  research  allows  administrators  to  define  the  desired  behavior  of  the  participants  in  the  system,  both  human  and 
software,  with  one  collection  of  well  defined  policies.  As  a  result,  a  single  set  of  tools  for  the  definition,  analysis, 
control,  and  monitoring  of  policy  can  be  used  to  implement  access  control,  service  configuration,  and  service 
delivery  prioritization. 


Eccles,  D.  W.,  Feltovich,  P.  J.,  &  Hoffman,  R.  R.  (2003).  Modeling  Terrain  Analysis  Expertise  to  Support 
Command  and  Control  (C2)  in  the  Field.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003. 
Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland 
University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

This  report  describes:  (1)  a  process  for  capturing  the  knowledge  of  military  experts,  and  (2)  the  use  of  models  of  that 
expert  knowledge  in  a  revolutionary  re-design  of  military  field  manuals  and  other  documents,  transforming  them 
into  a  new  type  of  knowledge  resource  that  "takes  the  expert  into  the  field."  The  project  focused  on  expertise  at 
terrain  analysis  after  it  was  realized  that  coping  with  terrain  is  the  mostfrequently  cited  source  of  difficulty  in  C2 
activities.  The  development  process  involved  identifying  experts  at  the  US  Army  Corps  of  Engineers  and  then 
conducting  Cognitive  Task  Analysis  (CTA)  procedures  in  order  to  elicit  their  knowledge.  What  resulted  was  a 
corpus  of  thousands  of  propositions  concerning  terrain  (soils,  rock  types,  drainage  patterns,  etc.),  referred  to  as  the 
Terrain  Analysis  Data  Base  (Hoffman,  1984).  The  TADB  has  existed  only  in  the  form  of  a  text  document.  Our  work 
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for  the  CTA  Program  involved  a  new  process  called  "knowledge  recovery,"  in  which  expertise  that  is  latent  in  text 
formats  can  be  recovered  and  placed  into  new  formats  making  that  expertise  more  accessible  and  usable.  Using  the 
CmapTools  software,  we  created  over  100  Concept  Maps,  which  included  all  of  the  propositions  in  the  TADB.  Next, 
the  textual  information  in  the  "Army  Field  Manual  for  Terrain  Analysis"  (Department  of  the  Army,  1990)  was  cut 
into  meaningful  units  and  these  were  hyperlinked  to  the  concept  nodes  in  the  Concept  Maps.  The  resulting 
knowledge  model,  "Representation  of  Conceptual  Knowledge-Terrain  Analysis"  (ROCK-TA)  represents  expert- 
level  knowledge  about  terrain,  recaptured  from  older  text  documents  and  placed  into  an  electronic  form  that  can  be 
taken  into  the  battlefield  context.  Used  in  conjunction  with  GIS-based  map  information,  ROCK-TA  can  be  used  by 
commanders  at  the  "Unit  of  Action"  level,  or  planners  at  the  Battalion  level  who  are  engaged  in  intelligence 
preparation  of  the  battlefield.  The  system  should  support  the  evaluation  of  terrain  for  such  things  as  trafficability, 
line  of  sight,  cover  &  concealment,  and  engineering  properties.  Furthermore,  ROCK-TA  is  a  living  e-document  that 
can  be  continually  updated  with  new  information  about  terrain  and  expert  reasoning  about  terrain.  We  are  currently 
exploring  the  possibility  of  using  the  Concept  Map  format  to  re-design  Army  Operation  Orders,  also  to  make  them 
more  useful  in  the  battle  context  than  traditional  text  by  making  the  information  they  contain  more  readily 
accessible. 


Eckroth,  J.,  Reddy,  D.,  Josephson,  J.  R.,  Chellappa,  R.,  &  Miller,  T.  N.  (2009).  Chapter  2.  From  Background 
Subtraction  to  Threat  Detection  in  Automated  Video  Surveillance.  In  P.  McDermott  and  L.  Allender  (Eds.), 
Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  I.  Collecting, 
Processing  and  Distributing  Battlefield  Information:  48hrBooks  by  Alion  Science  &  Technology  Corporation  under 
contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and 
Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative 
Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the 
Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003. 
http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 
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Summary  abstract: 

As  video  cameras  and  other  sensors  become  cheaper  and  easier  to  network,  sensor  networks  become  increasingly 
attractive  means  to  acquire  useful  information  for  military  operations,  such  as  video  surveillance  for  facilities 
protection,  and  "persistent  surveillance"  to  maintain  sensory  contact  with  targets  of  interest.  However,  without 
assistance  from  automation,  humans  will  be  overloaded  by  information,  and  unable  to  use  it  effectively. 

"What  information  consumes  is  rather  obvious:  it  consumes  the  attention  of  its  recipients.  Hence  a  wealth  of 
information  creates  a  poverty  of  attention,  and  a  need  to  allocate  that  attention  efficiently  among  the  overabundance 
of  information  sources  that  might  consume  it." 

It  is  difficult  for  humans  to  vigilantly  monitor  a  large  number  of  video  feeds  for  extended  periods  without  fatigue,  or 
complacency,  especially  if  they  are  asked  to  recognize  the  significance  of  rare  events  in  a  complex  stream  of  events. 
Moreover,  sometimes  events  of  interest  cannot  be  recognized  simply  from  the  video  without  knowing  where  a 
specific  camera  is  pointing,  at  a  restricted  area,  for  example.  Keeping  track  of  the  location  and  significance  of  a 
camera's  field  of  view  imposes  an  additional  cognitive  burden.  The  cognitive  burden  is  even  greater  if  recognizing 
events  of  interest  requires  tracking  entities  as  they  become  visible  in  different  cameras,  and  accessing  mental  maps 
to  understand  the  significance  of  motion  paths. 

Automation  can  help.  It  can  potentially  provide  users  with  alerts  based  on  recognizing  indicators  of  threatening 
behavior,  including  behavior  that  cannot  be  detected  with  a  single  camera.  For  example,  the  movement  of  an  entity 
from  place  to  place  in  relation  to  the  map  might  show  a  pattern  indicative  of  scouting  the  perimeter  of  a  facility, 
where  no  single  camera  view  shows  anything  suspicious.  Many  unsolved  technical  problems  remain,  however, 
including  problems  about  how  to  extract  needed  information  from  video  imagery,  and  problems  of  how  to  process 
acquired  information  to  climb  the  levels  of  abstraction  from  individual  camera-centered  frameworks  to  a  world- 
centered  framework,  and  from  motion  description  ("Object  05604  moved  along  path  P52  from  location  L12  at  time 
T  55  to  L13  at  T  66"')  to  behavior  description  (,,05604  proceeded  slowly  north  on  road  R3  to  the  intersection  with 
R7,  turned  East  on  R7  ....  ")  to  recognition  of  indicators  of  threat  ("°5604  slowly  circled  the  facility:').  . 

Abundant  video  also  threatens  to  overload  communications  and  storage  systems.  Typically,  the  sensed  bits  are 
highly  redundant  for  the  task  to  be  performed.  For  example,  when  it  is  important  to  track  a  moving  object,  the 
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important  information  is  which  part  ofthe  image  is  changing  continuously.  To  understand  which  region  is  relevant, 
subtraction  of  the  unchanging  background  is  needed,  and  if  it  can  be  performed  at  the  sensor  itself,  significantly 
fewer  bits  need  to  be  transmitted.  Accordingly,  a  new  framework  can  be  envisaged  in  which  the  traditional  ways  of 
sensing  an  image  (in  rectangular  pixels)  are  replaced  by  sensing  the  image  information  in  a  compact  form 
("compressed  sensing"),  and  then  reconstructing  the  required  image,  as  needed,  from  this  compact  information. 

This  paper  describes  some  elements  of  recent  progress  in  technology  for  video  surveillance  that  specifically  address 
methods  for  background  subtraction  to  detect  changes  from  frame  to  frame  in  video,  tracking  of  viewed  objects  in 
camera-centered  "image  space,"  tracking  of  objects  in  world  oriented  "object  space"  using  information  from 
multiple  cameras,  and  methods  for  "climbing"  levels  of  abstraction  in  descriptions  of  behavior. 


Elm,  W.,  Potter,  S.,  Tittle,  J.,  Woods,  D.,  Grossman,  J.,  &  Patterson,  E.  (2005).  Finding  Decision  Support 
Requirements  for  Effective  Intelligence  Analysis  Tools.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Fluman 
Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Within  ARDA's  GI2Vis  program,  we  developed  a  unique  framework  for  the  definition  of  decision  support 
requirements  for  intelligence  analysis  tools.  This  framework,  based  on  a  first-of-a-kind  integration  of  a  model  of 
inferential  analysis  and  principles  for  designing  effective  human-computer  teams  from  Cognitive  Systems 
Engineering,  has  defined  the  essential  support  functions  to  be  provided  to  the  intelligence  analyst(s).  This  model  has 
proven  to  be  extremely  useful  in  assessing  the  support  provided  by  a  large  set  of  visualization  tools.  This  assessment 
has  identified  clusters  of  support  functions  that  are  addressed  by  many  tools  as  well  as  key  missing  support 
functions.  In  this  way,  the  Support  Function  Model  has  been  used  to  identify  gaps  in  the  support  function  coverage 
of  existing  tools.  This  can  serve  as  a  valuable  focusing  mechanism  for  future  design  and  development  efforts.  In 
addition,  we  believe  this  would  be  a  useful  mechanism  to  enhance  cross-discussions  among  research  teams  involved 
in  Cognitive  Task  Analysis  efforts  within  the  Intelligence  Community.  Having  others  integrate  their  analytic  results 
with  this  framework  would  provide  the  mechanism  for  expansion  of  this  model  to  become  a  more  robust  tool  and 
have  an  even  greater  impact  on  the  Intelligence  Community. 


Elm,  W.  C.,  Woods,  D.  D.,  Bennett,  K.,  Bisantz,  A.,  Eggleston,  R.,  &  Mitchell,  C.  (2002).  Behind  the  Curtain:  The 
Cognitive  Tasks  Behind  the  Visualizations  (Representational  Types  for  Effective  Visualization  Design)  (Discussion 
Panel).  In  In  Proceedings  of  46th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Baltimore,  MD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  panel  includes  participants  from  academia  and  industry  who  have  each  made  significant  contributions  to  the 
design  of  effective  visualizations  to  support  decision  making  in  a  variety  of  domains.  Panel  members  will  offer  an 
example  of  an  innovative  decision  support  visualization  concept  and  discuss  the  underlying  cognitive  demands  it  is 
meant  to  support  as  well  as  any  artifacts  they  used  in  its  development.  Both  the  nature  of  the  visualization  itself,  and 
the  linkage  to  the  processes  'behind  the  curtain'  will  be  presented.  Panelists  will  discuss  the  various  techniques  used 
and  the  pro's  and  con's  of  each. 


Endsley,  M.,  &  Hoffman,  R.  R.  (2003).  The  Sacagawea  Principle.  IEEE  Intelligent  Systems,  November,  80-85. 
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Summary  abstract: 

Many  software  tools  and  systems  restrict  the  availability  of  information  and  make  information  integration  and 
exploration  difficult.  Poorly  designed  tools  are  often  brittle,  because  they  prescribe  task  sequences.  But  in  complex 
sociotechnical  contexts,  workers  do  not  perform  tasks;  they  engage  in  knowledge-driven,  context-sensitive  choices 
from  among  action  sequence  alternatives  in  order  to  achieve  goals.  So,  good  tools  must  be  flexible — they  must 
provide  the  information  that  workers  need  to  generate  appropriate  action  sequences  by  which  they  can  achieve  the 
same  goal  in  different  situations.  Adapted  from  the  writings  of  Donald  Norman  is  a  principle  we  call  the  Sacagawea 
Principle: 

Human-centered  computational  tools  need  to  support  active  organization  of  information,  active  search  for 
information,  active  exploration  of  information,  reflection  on  the  meaning  of  information,  and  evaluation  and  choice 
among  action  sequence  alternatives. 

Context-conditional  variation  includes  variation  due  to  the  worker — each  worker  has  his  or  her  own  needs,  entailing 
different  requirements  and  constraints.  This  implies  that  individuals  should  be  able  to  choose  different  trajectories  to 
achieve  the  desired  outcome  in  different  ways.  A  good  tool  gives  users  discretion  to  generate  various  action 
sequences  and  express  their  preferences. 

As  with  many  HCC  principles,  we  have  named  this  one  after  a  person  to  give  it  a  concrete  and  meaningful  label. 
Sacagawea  served  as  a  guide,  without  whose  help  the  Lewis  and  Clark  expedition  might  not  have  achieved  the 
successes  it  did.  The  name  is  also  somewhat  ironic,  because  Sacagawea  was,  for  part  of  her  life,  a  captured  slave. 
The  theme  of  machines  and  robots  as  slaves  is  arguably  the  oldest  in  the  robotics  literature,  and  it  is  still  often  used 
as  a  metaphor  to  describe  the  tools  people  use  to  accomplish  their  work.  In  this  essay,  we  explore  an  approach  for 
fulfilling  the  Sacagawea  Principle  in  system  design — an  approach  based  on  empirical  study  of  the  way  in  which 
people  process  their  environments  in  complex  worlds. 


Endsley,  M.  R.  (2001).  Designing  for  Situation  Awareness  in  Complex  Systems.  In  In  Proceedings  of  Second 
International  Workshop  on  Symbiosis  of  Humans,  Artifacts  and  Environment,  pp.  176-190.  Kyoto,  Japan:  Program 
of  the  Japan  Society  for  the  Promotion  of  Science. 

Key  Words: 
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Summary  abstract: 

We  are  living  in  what  has  been  termed  the  "information  age".  In  many  domains,  this  has  meant  a  huge  increase  in 
systems,  displays  and  technologies.  From  voice  control  to  sophisticated  line  of  sight  head  mounted  displays,  almost 
anything  is  possible  in  today's  world,  but  too  much  is  proving  to  be  as  big  a  challenge  as  too  little  once  was.  The 
problem  is  no  longer  lack  of  information,  but  finding  what  is  needed  when  it  is  needed. 

This  problem  is  not  isolated  to  cockpits  or  power  plants.  All  around  us,  signs  of  this  change  are  present.  Whether 
you  are  working  on  the  shop  floor,  in  the  world  of  business,  or  just  trying  to  purchase  a  new  computer  for  your 
home,  the  dizzying  pace  of  technological  change  and  vast  amount  of  information  present  can  be  daunting.  We  are 
constantly  being  barraged  with  information  through  TV,  radio,  mailings  and  hundreds  of  magazines  and  journals. 
Within  our  companies,  reports  and  forms  have  multiplied  and  every  aspect  of  the  business  is  recorded  somewhere. 
Bringing  all  of  this  information  together  in  a  form  that  is  manageable  is  quite  a  challenge.  There  is  simply  more 
information  than  anyone  can  handle. 


Endsley,  M.  R.  (2003).  Breaking  through  the  Data  Glut:  Bringing  to  Light  the  Meaning  of  Information.  HSIAC 
Gateway.  XIII  (4),  p.  7. 
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Summary  abstract: 

Today,  it  is  possible  to  gather  and  transTnit  vast  quantities  of  data.  However,  this  unprecedented  access  to  data  has 
failed  to  produce  hoped  for  leaps  in  understanding  because  there  is  a  huge  gap  between  the  glut  of  data  produced 
and  disseminated  and  users’  abilities  find  and  process  the  information  they  reaHy  want  amongst  all  that  is  possible. 
This  gap  challenges  many  in  positions  crucial  to  security  and  homeland  defense — the  intelligence  analyst  who  may 
read  thousands  of  messages  to  unearth  a  terrorist  plots,  an  American  soldier  who  must  integrate  and  coordinate  the 
activities  of  widely  distributed  units  in  a  foreign  country  surrounded  by  hostile  combatants  as  well  as  civilians,  and 
millions  who  need  to  be  able  to  detect  information  attacks  by  hackers.  Just  as  information  is  a  tool  in  our  arsenal,  it 
also  serves  as  a  tool  of  those  who  seek  to  undermine  the  U.S.  and  its  institutions. 


Endsley,  M.  R.  (2003).  Macro-Cognition:  Directions  at  the  Crossroads.  In  In  Proceedings  of  6th  International 
Conference  on  Naturalistic  Decision  Making.  Pensacola  Beach,  FL. 
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Summary  abstract: 

Macrocognition  has  been  defined  as  a  description  of  cognitive  functions  in  natural  decision  making  settings 
(Cacciabue  &  Hollnagel,  1995).  As  such,  it  has  much  in  common  with  other  terms  or  bodies  of  study  including 
situated  cognition,  naturalistic  decision  making  and  cognitive  engineering.  All  of  these  approaches  have  recognized 
the  importance  of  context  in  understanding  human  cognition  and  behaviors,  mostly  as  a  rejection  of  the  sterile, 
laboratory-based  cognitive  research  of  more  traditional  psychology  endeavors.  As  such,  they  signify  a  different 
research  paradigm,  with  different  goals  and  objectives  from  microcognitive  approaches.  Understanding  the 
cognitive  processes,  behaviors  and  products  of  human  beings  requires  an  understanding  of  the  contexts  they  operate 
in  and  the  tools  and  technologies  available  to  them,  and  most-importantly,  how  those  factors  interact.  Cognition  does 
not  occur  in  a  vacuum,  but  rather  is  a  complex  product  of  the  brain’s  capabilities  and  its  environment. 

Moving  forward  in  the  development  of  a  science  and  practice  requires  more  than  descriptions  of  what  we  want  to 
study  and  why.  In  many  ways,  we  have  been  studying  these  phenomenon  for  decades.  Cognitive  engineering, 
naturalistic  decision  making,  situated  cognition,  macrocognition. ...  we  have  wandered  over  a  broad  landscape 
making  discoveries  along  the  way,  but  as  of  yet  it  has  been  a  rather  random  journey.  The  tools  of  progress  forging  a 
lasting  paradigm  remain  the  same  for  us  as  for  others.  According  to  Kuhn  (1996),  for  any  new  paradigm  to  be 
lasting  it  must  be  capable  of  predicting  different  outcome  than  its  predecessors  to  resolve  observed  anomalies,  and  in 
so  doing  must  be  able  to  extend  those  predictions  to  new  areas  for  which  no  precedent  exists.  If  we  cannot,  the 
paradigm  fails  and  people  return  to  previous  paradigms. 

Setting  ourselves  on  a  productive  course,  therefore,  requires  that  as  a  field  we  develop  sound  research 
methodologies,  tools  and  approaches.  We  must  move  beyond  merely  observing  human  behaviors  and  making 
theories  about  cognitive  processes,  a  necessarily  subjective  process.  It  is  necessary  that  methods  be  developed  for 
evaluating  and  comparing  competing  hypotheses;  for  moving  from  descriptive  theories  to  those  that  are  predictive. 
Only  then  will  this  field  form  a  lasting  body  of  science  that  will  be  useful. 

Just  as  cognition  does  not  exist  in  a  vacuum,  neither  does  research;  it’s  outcome  and  methods  are  largely  dependant 
upon  its  goals.  The  ultimate  aim  of  our  profession  is  the  better  design  of  tools,  technologies  and  organizations  to 
support  cognitive  work.  Achieving  this  goal  necessitates  that  we  tackle  the  really  hard  problems.  Drawing 
predictive  generalizations  from  observations  of  processes  that  are,  by  definition,  affected  by  context  is  a  daunting 
challenge,  but  a  necessary  one.  Finding  ways  to  measure  and  assess  human  cognition  in  order  to  validate  and 
compare  theories  and  hypotheses  in  new  settings  is  equally  challenging  but  necessary.  The  course  we  must  take  is 
difficult  but  clear.  It  remains  for  us  to  successfully  meet  that  challenge. 


Endsley,  M.  R.  (2006).  Chapter  36.  Expertise  and  Situation  Awareness.  In  A.  Ericsson,  N.  Charness,  R.  Hoffman,  & 
P.  Feltovich  (Eds.),  Cambridge  Handbook  on  Expertise  and  Expert  Performance.  Cambridge:  Cambridge  University 
Press. 
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Summary  abstract: 

In  thinking  about  expertise,  we  often  focus  on  skilled  physical  performance  (e.g.,  the  world-class  tennis  player  or 
gymnast)  or  skilled  decision  making  (e.g.,  the  chess  grandmaster).  In  addition  to  these  aspects  of  performance, 
however,  situation  awareness  (SA),  an  up-to-date  understanding  of  the  world  around  them,  forms  a  critical 
cornerstone  for  expertise  in  most  domains,  from  driving  to  aviation  to  military  operations  to  medical  practice.  The 
characteristics  that  allow  people  to  develop  high  levels  of  SA  often  develop  silently  alongside  more  observable 
features  like  skilled  physical  performance,  even  in  tasks  such  as  sports  that  are  considered  primarily  physical  in 
nature.  Take  for  example  the  following  excerpt  from  a  magazine  story  about  Wayne  Gretzky,  an  all-time  leading 
hockey  scorer  who  set  or  tied  49  different  National  Hockey  League  records,  including  most  goals,  most  points,  and 
most  assists. 

Gretzky  doesn't  look  like  a  hockey  player.  ...  His  shot  is  only  average  -  or;  nowadays,  below  average  .  ..  Gretzky's 
gift,  his  genius  even,  is  for  seeing  .  ..  To  most  fans,  and  sometimes  even  to  the  players  on  the  ice,  hockey  frequently 
looks  like  chaos:  sticks  flailing,  bodies  falling,  the  puck  ricocheting  just  out  of  reach.  But  amid  the  mayhem, 
Gretzky  can  discern  the  game's  underlying  pattern  and  flow,  and  anticipate  what's  going  to  happen  faster  and  in 
more  detail  than  anyone  else  in  the  building.  Several  times  during  a  game  you'll  see  him  making  what  seem  to  be 
aimless  circles  on  the  other  side  of  the  rink  from  the  traffic,  and  then,  as  if  answering  a  signal,  he'll  dart  ahead  to  a 
spot  where,  an  instant  later;  the  puck  turns  up.  (McGrath,  1997) 

Although  undoubtedly  Wayne  Gretzky  possessed  the  important  physical  skills  associated  with  the  sport  of  hockey, 
this  article  points  out  that  the  critical  attribute  that  placed  him  head  and  shoulders  above  his  contemporaries  was 
mental  -  his  ability  to  understand  what  was  happening  in  the  game  and  to  anticipate  where  the  puck  would  be.  This 
superior  situation  awareness  allowed  him  to  be  "ahead  of  the  game"  and  outmatch  bigger,  faster,  and  better  players. 
Similar  stories  can  be  found  in  other  sports  (for  example,  football,  basketball,  or  tennis  -  see  also  Hodges  et  al., 
Chapter  27)  in  which  anticipating  the  actions  of  one's  teammates,  one's  opponents,  and  where  the  ball  is  going  are 
key  to  effective  individual  and  team  performance.  The  importance  of  situation  awareness  can  be  found  even  in  a 
relatively  straightforward  sport  such  as  golf,  which  at  first  glance  might  appear  to  be  only  a  matter  of  a  mechanical 
match  between  the  golf  swing  and  the  distance  and  bearing  to  the  hole.  Even  in  this  sport  situation  awareness  has  a 
role,  however.  Expert  players  will  walk  the  course  ahead  of  time  to  take  in  key  situational  features  that  make  one 
course  play  differently  from  another.  In  the  2004  Masters  Golf  Tournament,  Phil  Mickelson  won  by  sinking  a  putt 
on  the  last  hole.  Credited  as  critical  to  that  putt  was  the  fact  that  the  player  immediately  previous  to  Mickelson 
putted  from  almost  the  exact  same  location.  This  allowed  Phil  Mickelson  to  "read"  the  hole  -  observing  the  winds 
and  very  slight  variations  of  the  grass  and  grades  of  the  slope  between  the  ball  and  the  hole.  He  consciously  worked 
to  develop  the  best  SA  possible  before  taking  his  swing. 

Situation  awareness  plays  an  even  more  important  role  in  other  domains,  such  as  military  operations,  piloting,  or  air 
traffic  control,  where  there  are  many  factors  to  keep  track  of  and  these  factors  can  change  quickly  and  interact  in 
complex  ways.  Effective  decision  making  depends  on  high  levels  of  SA,  and  thus  so  does  effective  performance.  In 
this  chapter  I  will  discuss  the  ways  in  which  SA  is  critical  to  expert  performance  and  the  factors  that  allow  it  to 
improve  with  the  development  of  expertise  in  a  domain.  Studies  from  several  different  arenas  will  be  presented  to 
highlight  the  many  difficulties  that  novices  have  in  developing  good  SA  and  to  show  how  SA  improves  as 
performers  develop  expertise. 


Endsley,  M.  R.  (2009).  Chapter  20.  Situation  Awareness.  In  G.  Salvendy  (Ed.),  Handbook  of  Human  Factors  and 
Ergonomics.  London:  John  Wiley  &  Sons,  Inc. 
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Summary  abstract: 

As  we  move  into  the  twenty-first  century,  the  biggest  challenge  within  most  industries  and  the  most  likely  cause  of 
an  accident  receives  the  label  of  human  error.  This  is  a  most  misleading  lerm,  however,  thal  has  done  much  to  sweep 
the  real  problems  under  the  rug.  It  implies  that  people  are  merely  careless  or  poorly  trained  or  somehow  not  very 
reliable  in  general.  In  fact,  in  the  vast  majority  of  these  incidents  the  human  operator  was  striving  against  significant 
challenges.  On  a  day-to-day  basis,  they  cope  with  hugely  demanding  complex  systems.  They  face  both  data  overload 
and  the  challenge  of  working  with  a  complex  system.  They  are  drilled  with  long  lists  of  procedures  and  checklists 
designed  to  cope  with  some  of  these  difficulties,  but  from  time  to  time  they  are  apt  to  fail.  Industry's  typical 
response  to  such  failures  has  been  more  procedures  and  more  systems,  but  unfortunately,  this  only  adds  to  the 
complexity  of  the  system.  In  reality,  the  person  is  not  the  cause  of  these  errors  but  is  the  final  dumping  ground  for 
the  inherent  problems  and  difficulties  in  the  technologies  we  have  created.  The  operator  is  usually  the  one  who  must 
bring  it  all  together  and  overcome  whatever  failures  and  inefficiencies  exist  in  the  system. 


Endsley,  M.  R.,  Bolstad,  C.  A.,  Jones,  D.  G.,  &  Riley,  J.  M.  (2003).  Situation  Awareness  Oriented  Design:  From 
User's  Cognitive  Requirements  to  Creating  Effective  Supporting  Technologies.  In  In  Proceedings  of  47th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Situation  awareness  is  a  fundamental  construct  driving  human  decision  making  in  complex,  dynamic  environments. 
By  creating  designs  that  enhance  an  operator's  awareness  of  what  is  happening  in  a  given  situation,  decision  making 
and  performance  can  improve  dramatically.  The  Situation  Awareness-Oriented  Design  process  provides  a  means  to 
improve  human  decision-making  and  performance  through  optimizing  situation  awareness.  This  method  has  been 
used  to  develop  and  evaluate  system  design  concepts  in  aviation,  medical  and  information  intelligence  operations.  It 
features  three  main  components:  SA  Requirements  Analysis,  SA-Oriented  Design  Principles,  and  SA  Measurement 
and  Validation.  This  design  process  is  user-centered,  and  derived  from  a  detailed  analysis  of  the  goals,  decisions  and 
situation  awareness  requirements  of  the  operator  derived  through  a  Cognitive  Task  Analysis  methodology  called 
Goal-Directed  Task  Analysis.  The  development  of  tool  suites  for  supporting  high  levels  of  situation  awareness  in 
military  command  and  control  are  presented  to  illustrate  the  use  of  the  SAOriented  Design  process  for  translating  the 
results  of  cognitive  task  analyses  into  to  usercentered  system  designs. 


Endsley,  M.  R.,  Bolte,  B.,  &  Jones,  D.  G.  (2003).  Designing  for  Situation  Awareness:  An  Approach  to  Human 
Centered  Design.  London:  Taylor  &  Francis. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book 

PART  ONE:  UNDERSTANDING  SITUATION  AWARENESS  IN  SYSTEM  DESIGN 

1.  User-centered  Design 

2.  What  is  Situation  Awareness? 

3.  SA  Demons:  The  Enemies  of  Situation  Awareness 

4.  The  Design  Process 

PART  TWO:  CREATING  SITUATION  AWARENESS-ORIENTED  DESIGNS 

5.  Determining  SA  Requirements 

6.  Principles  of  Designing  for  SA 

7.  Confidence  and  Uncertainty  in  SA  and  Decision  Making 

8.  Dealing  with  Complexity 

9.  Alarms,  Diagnosis,  and  SA 

10.  Automation  and  Situation  Awareness 

1 1 .  Designing  to  Support  SA  for  Multiple  and  Distributed  Operators 
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PART  THREE:  COMPLETING  THE  DESIGN  CYCLE 

12.  Evaluating  Design  Concepts  for  SA 

13.  Applying  SA-oriented  Design  to  Complex  Systems 

Appendix  A:  Goal-directed  Task  Analysis  for  Commercial  Airline  Pilots 

Summary  abstract: 

Preface 

'Our  Age  of  Anxiety  is,  in  great  part,  the  result  of  trying  to  do  today 's  jobs  with  yesterday's  tools.  .  -  Marshall 
McLuhan 

While  a  clear  understanding  of  one's  situation  is  undoubtedly  the  critical  trigger  that  allows  the  knowledge,  skills, 
and  creativity  of  the  human  mind  to  be  successfully  brought  to  bear  in  shaping  our  environment,  very  often  people 
must  work  uphill,  against  systems  and  technologies  that  block  rather  than  enhance  their  ability  to  ascertain  the 
information  they  need.  Knowledge  in  a  vacuum  is  meaningless.  Its  use  in  overcoming  human  problems  and 
achieving  human  goals  requires  the  successful  application  of  that  knowledge  in  ways  that  are  contextually 
appropriate.  Yet,  across  a  wide  variety  of  engineered  systems,  people  face  an  everwidening  information  gap-the  gulf 
between  the  data  that  is  available  and  the  information  that  they  really  need  to  know. 

This  book  addresses  the  information  gap  through  system  design.  It  presents  a  core  set  of  principles  and  a 
methodology  for  engineers  and  designers  who  are  seeking  to  nourish  the  situation  awareness  of  their  system's  users. 
Operators  of  power  plants,  aircraft,  automobiles,  ships,  command  and  control  centers  for  military  and  large-scale 
commercial  enterprises,  intelligence  operations,  medical  systems,  and  information  management  systems  are  all  in 
need  of  technologies  that  allow  people  to  effectively  manage  the  information  available  to  gain  a  high  level  of 
understanding  of  what  is  happening.  They  need  systems  designed  to  support  situation  awareness. 

We  are  indebted  to  many  people.  First,  we  are  grateful  for  the  support  and  patience  of  our  families  during  the 
writing  of  this  book-may  it  lead  to  a  better  world  for  our  children.  We  are  thankful  for  the  support  of  many  of  our 
colleagues,  whose  dedicated  research  has  created  the  foundation  upon  which  this  book  is  based.  Finally,  we  are 
indebted  to  the  countless  number  of  pilots,  air  traffic  controllers,  soldiers,  and  system  operators  who  have 
volunteered  their  insights,  experiences,  time,  and  expertise  in  the  hopes  of  bettering  the  tools  of  their  profession.  We 
also  appreciate  and  acknowledge  the  support  of  our  sponsors,  the  U.S.  Army  Research  Laboratory  Collaborative 
Technology  Alliance  and  the  NASA  Langley  Research  Center,  without  whose  assistance  this  book  would  not  have 
been  possible. 

Although  we  are  committed  to  scientific  truth,  there  comes  a  point  where  this  truth  is  not  enough,  where  the 
application  of  truth  to  human  objectives  comes  into  play.  Once  we  start  to  think  in  terms  of  utility,  we  must  leave  the 
placid  environment  of  the  laboratory,  take  off  our  white  coats,  and  roll  up  our  sleeves  ....  "  Samuel  Florman-  'The 
Civilized  Engineer' 


Endsley,  M.  R.,  &  Jones,  D.  G.  (2001).  Disruptions,  Interruptions  and  Information  Attack:  Impact  on  Situation 
Awareness  and  Decision  Making.  In  In  Proceedings  of  45th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Minneapolis,  MN  (pp.  63-67). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  presents  a  model  of  depicting  the  ways  in  which  disruptions,  interruptions  and  information  attack  can 
effect  situation  awareness  and  decision  making  in  a  variety  of  contexts.  Those  effected  in  military  aviation  include 
not  just  those  in  the  cockpit,  but  also  forward  air  controllers,  ground  based  air  traffic  controllers  and  those  in 
intelligence  and  support  functions.  The  model  incorporates  the  ways  in  which  information  attacks  can  effectively 
disrupt  human  decision  making  at  various  points  in  information  processing.  By  carefully  examining  not  just  what 
cues  might  depict  an  attack  to  information  systems,  but  also  how  human  observers  will  be  effected  by  such  cues, 
more  robust  systems  for  protecting  against  disruptions  and  information  attack  can  be  developed. 
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Endsley,  M.  R.,  &  Jones,  D.  G.  (2001).  Disruptions,  Interruptions,  and  Information  Attack:  Impact  on  Situation 
Awareness  and  Decision  Making.  In  In  Proceedings  of  45th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Minneapolis,  MN. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  presents  a  model  of  depicting  the  ways  in  which  disruptions,  interruptions  and  information  attack  can 
effect  sittmtion  awareness  and  decision  making  in  a  variety  of  contexts.  Those  effected  in  military  aviation  include 
not  just  those  in  the  cockpit,  but  also  forward  air  controllers,  ground  based  air  traffic  controllers  and  those  in 
intelligence  and  support  functions.  The  model  incorporates  the  ways  in  which  information  attacks  can  effectively 
disrupt  human  decision  making  at  various  points  in  information  processing.  By  carefully  examining  not  just  what 
cues  might  depict  an  attack  to  information  systems,  but  also  how  human  observers  will  be  effected  by  such  cues, 
more  robust  systems  for  protecting  against  disruptions  and  information  attack  can  be  developed. 


Endsley,  M.  R.,  Jones,  D.  G.,  Bolstad,  C.  A.,  &  Riley,  J.  M.  (2003).  Designing  to  Support  Situation  Awareness  in 
the  Objective  Force.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and 
Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University 
College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

The  SA-Oriented  Design  process  provides  a  means  to  improve  human  decision-making  and  performance  by 
optimizing  situation  awareness  (SA)  in  system  design.  It  provides  a  methodology  and  guidelines  for  overcoming  the 
system  complexity  and  data  overload  problems  experienced  in  many  systems  and  provides  a  means  of  supporting 
the  needs  of  teams  to  have  shared  situation  awareness  to  support  their  tasks.  In  this  paper  we  will  provide  an 
overview  of  this  process  and  show  its  application  to  the  development  of  display  tools  to  support  SA  in  the  Objective 
Force. 


Ericsson,  K.  A.,  Charness,  N.,  Feltovich,  P.  J.,  &  Hoffman,  R.  R.  (Eds.).  (2006).  Cambridge  Handbook  of  Expertise 
and  Expert  Performance.:  New  York:  Cambridge  University  Press. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book 

Summary  abstract: 

This  is  the  first  handbook  where  the  world's  foremost  "experts  on  expertise"  review  our  scientific  knowledge  on 
expertise  and  expert  performance  and  how  experts  may  differ  from  non-experts  in  terms  of  their  development, 
training,  reasoning,  knowledge,  social  support,  and  innate  talent.  Methods  are  described  for  the  study  of  experts' 
knowledge  and  their  performance  of  representative  tasks  from  their  domain  of  expertise.  The  development  of 
expertise  is  also  studied  by  retrospective  interviews  and  the  daily  lives  of  experts  are  studied  with  diaries,  In  15 
major  domains  of  expertise,  the  leading  researchers  summarize  our  knowledge  of  the  structure  and  acquisition  of 
expert  skill  and  knowledge  and  discuss  future  prospects.  General  issues  that  cut  across  most  domains  are  reviewed 
in  chapters  on  various  aspects  of  expertise,  such  as  general  and  practical  intelligence,  differences  in  brain  activity, 
self-regulated  learning,  deliberate  practice,  aging,  knowledge  management,  and  creativity. 
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Fan,  X.,  McNeese,  M.,  Sun,  B.,  Flanratty,  T.,  Laurel  Allender,  &  Yen,  J.  (2006).  FIuman-Agent  Collaboration  for 
Time  Stressed  Multi-Context  Decision  Making.  In  In  Proceedings  of  Collaborative  Technology  Alliance  (CTA) 
Advanced  Decision  Architectures  (ADA)  Sub  Group,  2006.  IEEE  Transactions  On  Systems,  Man,  And 
Cybernetics,  Part  A:  Systems  And  Humans,  2006  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD 
21005,  Human  Research  and  Engineering  Directorate.  http://agentlab.psu.edu/lab/publications/Fan_SMC2009.pdf. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  Human-Centered  Computing, 
Cognitive  Agents,  Context  Switching,  Naturalistic  Decision  Making 

Summary  abstract: 

Multi-context  team  decision  making  under  time  stress  is  an  extremely  challenging  issue  faced  by  various  real  world 
application  domains.  In  this  study  we  employ  an  experience-based  cognitive  agent  architecture  (R-CAST)  to  address 
the  informational  challenges  associated  with  military  command  and  control  (C2)  decision  making  teams,  the 
performance  of  which  can  be  significantly  affected  by  dynamic  context  switching  and  tasking  complexities.  Using 
context  switching  frequency  and  task  complexity  as  two  factors,  we  conducted  an  experiment  to  evaluate  whether 
the  use  of  R-CAST  agents  as  teammates  and  decision  aids  can  benefit  C2  decision  making  teams.  Members  from  a 
US  Army  ROTC  (Reserve  Officer  Training  Corps)  organization  were  randomly  recruited  as  human  participants. 
They  were  grouped  into  ten  Human-Human  teams  each  composed  of  two  participants  and  ten  Human- Agent  teams 
each  composed  of  one  participant  and  two  R-CAST  agents  as  teammates  and  decision  aids.  Statistical  inference  of 
the  experiment  results  indicates  that  R-CAST  agents  can  significantly  improve  the  performance  of  C2  teams  in 
multi-context  decision  making  under  varying  time-stressed  situations. 


Fan,  X.,  Oh,  S.,  McNeese,  M.,  Yen,  J.,  Cuevas,  H.,  Strater,  L.,  &  Endsley,  M.  R.  (2008).  The  Influence  of  Agent 
Reliability  on  Trust  in  Human- Agent  Collaboration.  In  In  Proceedings  of  European  Conference  on  Cognitive 
Ergonomics  (ECCE  2008),  Madeira,  Portugal. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Reliability,  Automation  usage  decisions,  Trust,  Agent 

Summary  abstract: 

Motivation  -  To  investigate  ways  to  support  humanautomation  teams  with  real-world,  imperfect  automation  where 
many  system  failures  are  the  result  of  systematic  failure. 

Research  approach  -  An  experimental  approach  was  used  to  investigate  how  variance  in  agent  reliability  may 
influence  human’s  trust  and  subsequent  reliance  on  agent’s  decision  aids.  Sixty  command  and  control  (C2)  teams, 
each  consisting  of  a  human  operator  and  two  cognitive  agents,  were  asked  to  detect  and  respond  to  battlefield  threats 
in  six  ten-minute  scenarios.  At  the  end  of  each  scenario,  participants  completed  the  SAGAT  queries,  followed  by  the 
NASA  TLX  queries. 

Findings/Design  -  Results  revealed  that  teams  with  experienced  human  operators  accepted  significantly  less 
inappropriate  recommendations  from  agents  than  teams  with  inexperienced  operators.  More  importantly,  the 
knowledge  of  agent’s  reliability  and  the  ratio  of  unreliable  tasks  have  significant  effects  on  human’s  trust,  as 
manifested  in  both  team  performance  and  human  operators’  rectification  of  inappropriate  recommendations  from 
agents. 

Originality /Value  -  It  represents  an  important  step  toward  uncovering  the  nature  of  human  trust  in  humanagent 
collaboration. 

Take  away  message  -  This  research  has  shown  that  given  even  minimal  basis  for  understanding  when  the  operator 
should  and  should  not  trust  the  agent  recommendations  allows  operators  to  make  better  AUDs,  to  have  better 
situation  awareness  on  the  critical  issues  associated  with  automation  error,  and  to  establish  better  trust  in  intelligent 
agents. 


Fan,  X.,  Sun,  B.,  Sun,  S.,  McNeese,  M.,  &  Yen,  J.  (2006).  Rpd-Enabled  Agents  Teaming  with  Human  for  Multi- 
Context  Decision  Making.  In  In  Proceedings  of  Fifth  International  Joint  Conference  on  Aunomous  Agents  and 
Multi-Agent  Systems  (AAMAS  06),  Hakodate,  Japan. 
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Categories  and  Subject  Descriptors 

1.2.11  [Arti  cial  Intelligence]:  Distributed  Arti  cial  ln- 

telligence|Intelligent  agents,  Multiagent  systems 

General  Terms 

Design,  Experimentation,  Human  Factors 
Keywords 

Cognitive  agent.  Human-centered  teamwork.  Naturalistic  de¬ 
cision  making,  Multiple  contexts,  Attention  switching 

Summary  abstract: 

Team  decision  making  under  stress  involving  multiple  con-  texts  is  an  extremely  challenging  issue  faced  by  various 
real  world  application  domains.  This  research  is  targeted  at  cou-  pling  cognitive  agent  technology  and  human- 
centered  team-  work  to  address  the  informational  challenges  associated  with  Command  and  Control  (C2)  teams  in 
contemporary  military  operations.  Two  sets  of  experiments,  each  with  various  set-  tings  of  context  switching 
frequencies  and  tasking  complex-  ities,  were  conducted.  To  ensure  that  the  human  subjects  were  familiar  with  the 
C2  context,  they  were  selected  from  US  Army  ROTC  (Reserve  0±cer  Training  Corps)  students.  Experiments  on  C2 
teams  that  involve  human  subjects  only  were  conducted  rst.  We  observed  the  decision  making  be-  havior  of  human 
subjects  and  incorporated  expert  behaviors  into  R-CAST|an  agent  architecture  built  upon  a  naturalis-  tic  decision 
making  model  that  captures  how  domain  experts  make  decisions  based  on  experiences  and  situational  similar-  ity 
recognition.  We  then  conducted  another  set  of  experi-  ments  with  R-CAST  agents  as  teammates  and  decision  aids 
for  human  subjects.  The  results  show  that  RPD-enabled  agents  can  signi  cantly  improve  the  tasking  capacity  of  C2 
teams  in  multi-context  decision  making  under  stress.  It  also  suggests  that  higher  demand  situations  require  more 
com-  petent  teammates. 


Fan,  X.,  Sun,  S.,  McNeese,  M.,  &  Yen,  J.  (2005).  Extending  Recognition-Primed  Decision  Model  for  Human-Agent 
Collaboration.  In  In  Proceedings  of  the  Fourth  International  Joint  Conference  on  Autonomous  Agents  and  Multi 
Agent  Systems  (AAMAS'05),  July  2005. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Categories  and  Subject  Descriptors 
1.2  [Arti  cial  Intelligence]:  General|Cognitive  simula¬ 
tion;  1.2.1 1  [Arti  cial  Intelligence] :  Distributed  Arti  cial 
lntelligence|Intelligent  agents,  Multiagent  systems 
General  Terms 

Design,  Experimentation,  Human  Factors 
Keywords 

Human-centered  teamwork.  Human-agent  collaboration.  Nat¬ 
uralistic  decision  making.  Recognition  re  nement,  Shared 
situation  awareness 

Summary  abstract: 

There  has  been  much  research  investigating  team  cogni-  tion,  naturalistic  decision  making,  and  collaborative  tech¬ 
nology  as  it  relates  to  real  world,  complex  domains  ofprac-  tice.  However,  there  has  been  limited  work  in 
incorporat-  ing  naturalistic  decision  making  models  for  supporting  dis-  tributed  team  decision  making.  The  aim  of 
this  research  is  to  support  human  decision  making  teams  using  cognitive  agents  empowered  by  a  collaborative 
Recognition-Primed  Decision  model.  In  this  paper,  we  rst  describe  an  RPD-  enabled  agent  architecture  (R-CAST), 
in  which  we  have  im-  plemented  an  internal  mechanism  of  decision-making  adap-  tation  based  on  collaborative 
expectancy  monitoring,  and  an  information  exchange  mechanism  driven  by  relevant  cue  analysis.  We  have 
evaluated  R-CAST  agents  in  a  real-time  simulation  environment,  feeding  teams  with  frequent  decision-  making 
tasks  under  diUcrent  tempo  situations.  While  the  result  conforms  to  psychological  ndings  that  human  team 
members  are  extremely  sensitive  to  their  workload  in  high-  tempo  situations,  it  clearly  indicates  that  human  teams, 
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when  supported  by  R-CAST  agents,  can  perform  better  in  the  sense  that  they  can  maintain  team  performance  at 
acceptable  levels  in  high  time  pressure  situations. 


Fan,  X.,  Sun,  S.,  Sun,  B.,  Airy,  G.,  McNeese,  M.,  Yen,  J.,  Flanratty,  T.,  &  Dumer,  J.  (2005).  Collaborative  Rpd- 
Enabled  Agents  Assisting  the  Three-Block  Challenge  in  Command  and  Control  in  Complex  and  Urban  Terrain.  In 
In  Proceedings  of  Behavior  Representation  in  Modeling  and  Simulation  (BRIMS). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Collaborative  Agents,  Recognition-Primed  Decision  Making,  Human-Centered  Teamwork,  Contexts 

Summary  abstract: 

One  of  the  challenging  issues  in  the  domain  of  C2  in  complex  and  urban  terrain  is  how  to  assist  human  combat  staff 
to  effectively  collaborate  and  make  decisions  under  multiple  contexts,  and  to  help  people  switch  their  attentions  to 
the  most  urgent  decision  tasks.  This  paper  describes  a  comprehensive  approach  that  uses  RPD-enabled  agents  to 
support  timely  decision-making  under  multiple  contexts.  We  introduce  the  R-CAST  decision  support  system  and 
report  our  effort  of  using  the  R-CAST  agent  architecture  to  assist  human  combat  staff  in  dealing  with  information 
challenges  in  complex  and  urban  terrain.  The  simulation  demonstrates  that  by  modeling  the  information 
requirements  of  the  three  block  challenge  as  relevant  cues  and  expectancies,  R-CAST  agents  can  effectively  share 
relevant  information  for  complex  decision  situations. 


Fan,  X.,  &  Yen,  J.  (2006).  R-Cast:  Integrating  Team  Intelligence  for  Human-Centered  Teamwork.  In  In  Proceedings 
of  the  Twenty-Second  AAAI  Conference  on  Artificial  Intelligence,  July  22-26,  2007,  Vancouver,  British  Columbia. 
Published  by  The  AAAI  Press,  Menlo  Park,  California.  Sponsored  by  the  Collaborative  Technology  Alliance 
(CTA)  Advanced  Decision  Architectures  (ADA)  Sub  Group,  2006  U.S.  Army  Research  Laboratory,  Aberdeen 
Proving  Ground,  MD  21005,  Human  Research  and  Engineering  Directorate. 
http://www.aaai.org/Papers/AAAI/2007/AAAI07-243.pdf. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

Developing  human-centered  agent  architectures  requires  the  integral  consideration  of  architectural  flexibility, 
teamwork  adaptability,  and  context  reasoning  capability.  With  the  integration  of  various  forms  of  team  intelligence 
including  shared  teamwork  process  and  progress,  dynamic  context  management  and  information  dependency 
reasoning,  and  recognition-primed  collaborative  decision  mechanism,  R-CAST  offers  a  flexible  solution  to 
developing  cognitive  aids  for  the  support  of  human-centered  teamwork  in  information  and  knowledge  intensive 
domains.  In  this  paper,  we  present  the  key  features  of  R-CAST.  As  evidence  of  its  applications  in  complex  real- 
world  problems,  we  give  two  experimental  evaluations  of  R-CAST  as  teammates  and  decision  aids  of  human 
Command  and  Control  teams. 


Fan,  X.,  &  Yen,  J.  (2007).  R-Cast:  Integrating  Team  Intelligence  for  Human-Centered  Teamwork.  In  In  Proceedings 
of  Twenty-Second  AAAI  Conference  on  Artificial  Intelligence  (AAAI-07). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Developing  human-centered  agent  architectures  requires  the  integral  consideration  of  architectural  flexibility, 
teamwork  adaptability,  and  context  reasoning  capability.  With  the  integration  of  various  forms  of  team  intelligence 
including  shared  teamwork  process  and  progress,  dynamic  context  management  and  information  dependency 
reasoning,  and  recognition-primed  collaborative  decision  mechanism,  R-CAST  offers  a  flexible  solution  to 
developing  cognitive  aids  for  the  support  of  human-centered  teamwork  in  information  and  knowledge  intensive 
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domains.  In  this  paper,  we  present  the  key  features  of  R-CAST.  As  evidence  of  its  applications  in  complex  real- 
world  problems,  we  give  two  experimental  evaluations  of  R-CAST  as  teammates  and  decision  aids  of  human 
Command  and  Control  teams. 


Fan,  X.,  &  Yen,  J.  (2007).  Realistic  Cognitive  Load  Modeling  for  Enhancing  Shared  Mental  Models  in  Human- 
Agent  Collaboration.  In  In  Proceedings  of  Sixth  International  Joint  Conference  on  Autonomous  Agents  and  Multi- 
Agent  Systems  (AAMAS'07). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Categories  and  Subject  Descriptors 

1.2.11  [Artificial  Intelligence]:  Distributed  Artificial  Intelligence — 

Intelligent  agents,  Multiagent  systems 
General  Terms 

Design,  Experimentation,  Human  Factors 
Keywords 

Cognitive  modeling,  Human-centered  teamwork.  Shared  belief 
maps,  Multi-party  communication 

Summary  abstract: 

Human  team  members  often  develop  shared  expectations  to  predict  each  other’s  needs  and  coordinate  their 
behaviors.  In  this  paper  the  concept  “Shared  Belief  Map”  is  proposed  as  a  basis  for  developing  realistic  shared 
expectations  among  a  team  of  Human-Agent-Pairs  (HAPs).  The  establishment  of  shared  belief  maps  relies  on  inter¬ 
agent  information  sharing,  the  effectiveness  of  which  highly  depends  on  agents’  processing  loads  and  the 
instantaneous  cognitive  loads  of  their  human  partners.  We  investigate  HMM-based  cognitive  load  models  to 
facilitate  team  members  to  “share  the  right  information  with  the  right  party  at  the  right  time”.  The  shared  belief  map 
concept  and  the  cognitive/processing  load  models  have  been  implemented  in  a  cognitive  agent  architecture — 
SMMall.  A  series  of  experiments  were  conducted  to  evaluate  the  concept,  the  models,  and  their  impacts  on  the 
evolving  of  shared  mental  models  of  HAP  teams. 


Feltovich,  P.  J.,  Hoffman,  R.  R.,  &  Woods,  D.  D.  (2004).  Keeping  It  Too  Simple:  How  the  Reductive  Tendency 
Affects  Cognitive  Engineering.  IEEE  Intelligent  Systems,  90-95. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Certain  features  of  tasks  make  them  especially  difficult  for  humans.  These  constitute  leverage  points  for  applying 
intelligent  technologies,  but  there’s  a  flip  side.  Designing  complex  cognitive  systems  is  itself  a  tough  task. 
Cognitive  engineers  face  the  same  challenges  in  designing  systems  that  users  confront  in  working  the  tasks  that  the 
systems  are  intended  to  aid. 


Feltovich,  P.  J.,  Bradshaw,  J.  M.,  Clancey,  W.  J.,  Johnson,  M.,  &  Bunch,  L.  (2008).  Progress  Appraisal  as  a 
Challenging  Element  of  Coordination  in  Human  and  Machine  Joint  Activity.  In  In  Proceedings  of  Eighth  Annual 
International  Workshop  on  Engineering  Societies  in  the  Agents  World  (ESAW  2007).  Athens,  Greece. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Coordination,  culture,  human-agent-robotic  systems,  joint  activity,  ontology,  policy,  predictability,  regulation, 
teamwork,  progress  appraisal,  common  ground 
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Summary  abstract: 

Joint  activity,  as  we  define  it,  is  a  mutually  interdependent  endeavor  that  requires  a  degree  of  predictability  among 
participating  parties  so  that  they  can  coordinate.  Coordination,  in  turn,  sometimes  requires  the  parties  to  appraise  the 
state  of  progress  of  their  activities  so  that,  if  necessary,  they  can  adjust  their  actions  to  meet  coordination  needs  and 
communicate  their  status  to  others  as  appropriate.  Increasingly,  consequential  work  is  being  conducted  by  mixed 
groups  of  humans,  software  agents,  and  robots.  A  complaint  about  automation  has  been  that  it  often  does  not  have 
awareness  of  its  own  progress,  and  hence  can  give  no  warning  before  failure.  In  the  current  article,  we  address 
various  issues  associated  with  “progress  appraisal'’  and  the  challenges  it  poses  for  human-machine  systems.  We 
point  to  some  directions  for  possible  solutions. 


Feltovich,  P.  J.,  Bradshaw,  J.  M.,  Jeffers,  R.,  Suri,  N.,  &  Uszok,  A.  (2004).  Social  Order  and  Adaptability  in  Animal 
and  Human  Cultures  as  Analogues  for  Agent  Communities:  Toward  a  Policy-Based  Approach  In  In  Proceedings  of 
Fourth  International  Workshop  on  Engineering  Societies  in  the  Agents  World.  Imperial  College,  London. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  this  paper  we  discuss  some  of  the  ways  social  order  is  maintained  in  animal  and  human  realms,  with  the  goal  of 
enriching  our  thinking  about  mechanisms  that  might  be  employed  in  developing  similar  means  of  ordering 
communities  of  agents.  We  present  examples  from  our  current  work  in  humanagent  teamwork,  and  we  speculate 
about  some  new  directions  this  kind  of  research  might  take.  Since  communities  also  need  to  change  over  time  to 
cope  with  changing  circumstances,  we  also  speculate  on  means  that  regulatory  bodies  can  use  to  adapt. 


Ferris,  T.,  Hameed,  S.,  Penfold,  R.,  &  Rao,  N.  (2007).  Tactons  in  Multitask  Environments:  The  Interaction  of 
Processing  Code  and  Modality.  In  In  Proceedings  of  51st  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Baltimore,  MD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Tactons,  or  vibrotactile  icons,  have  been  proposed  as  a  means  to  communicate  complex  concepts  to  users  and  to 
support  multitasking  in  environments  involving  numerous  visual  and/or  auditory  tasks  and  stimuli.  This  study 
investigated  the  role  of  processing  code  in  the  interpretation  of  tactons  while  performing  concurrent  visual  tasks  in 
such  environments.  Participants  decoded  tactons  composed  of  spatiotemporal  patterns  of  vibrations  .  requiring 
spatial  processing  .  and  interpreted  one  of  two  types  of  visual  task  stimuli .  requiring  either  spatial  or  categorical 
processing  .  in  a  driving  simulation.  Compared  to  singletask  performance,  there  was  a  significantly  larger  dual-task 
performance  decrement  when  the  tacton  task  was  paired  with  the  visual  task  requiring  spatial  (as  compared  to 
categorical)  processing.  The  findings  are  consistent  with  the  assertion  of  Multiple  Resource  Theory  that  interference 
between  concurrent  tasks  is  greater  when  these  tasks  involve  the  same  processing  code.  They  illustrate  how 
distributing  task-related  information  across  modalities  is  beneficial  but  not  sufficient  to  avoid  task  interference.  A 
direct  implication  of  the  findings  is  to  avoid  the  use  of  spatiotemporal  tactons  in  environments  which  rely  heavily  on 
spatial  processing  resources,  such  as  car  cockpits  or  flight  decks. 


Ferris,  T.,  Hameed,  S.,  &  Sarter,  N.  (2009).  Tactile  Displays  for  Multitask  Environments:  The  Role  of  Concurrent 
Task  Processing  Code.  In  In  Proceedings  of  World  Haptics  2009  Conference.  Salt  Lake  City,  UT.  BEST  PAPER 
AWARD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
KEYWORDS:  Tacton  Displays,  Multiple  Resource  Theory,  Driving 
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INDEX  TERMS:  H.1.2  [Models  and  Principles]:  User/Machine 
Systems — Human  Information  Processing;  H.5.2  [Information 
Interfaces  and  Presentation]:  User  Interfaces — Haptic  I/O 

Summary  abstract: 

The  distribution  of  tasks  and  stimuli  across  multiple  modalities  has  been  proposed  as  a  means  to  support 
multitasking  in  data-rich  environments.  In  particular,  the  tactile  channel  and,  more  specifically,  communication  via 
the  use  of  tactile/haptic  icons  have  received  considerable  interest  in  recent  years.  Past  research  has  examined 
primarily  the  impact  of  concurrent  task  modality  on  the  effectiveness  of  tactile  information  presentation.  In  contrast, 
the  present  study  investigates  to  what  extent  the  interpretation  of  complex  tactile  patterns  -  “tactons”  -  is  affected  by 
another  attribute  of  information:  the  processing  code  of  concurrent  tasks.  Participants  decoded  tactons  composed  of 
temporal  patterns  of  vibrations  (categorical  data)  -  and  concurrently  interpreted  one  of  two  types  of  visual  task 
stimuli  -  requiring  either  spatial  or  categorical  processing  -  in  a  driving  simulation.  Compared  to  single-task 
performance,  both  dual-task  conditions  showed  a  performance  decrement.  As  predicted  by  Multiple  Resource 
Theory,  this  decrement  was  significantly  larger  when  the  tacton  task  was  paired  with  the  visual  task  requiring 
categorical  (as  compared  to  spatial)  processing.  The  findings  from  this  study  can  serve  as  input  to  multidimensional 
quantitative  models  of  timesharing  performance.  From  an  applied  perspective,  the  results  suggest  that  nonspatially- 
encoded  tactons  would  be  preferable  in  environments  which  rely  heavily  on  spatial  processing,  such  as  car  cockpits 
or  flight  decks. 


Ferris,  T.,  Penfold,  R.,  Hameed,  S.,  &  Sarter,  N.  (2006).  The  Implications  of  Crossmodal  Links  in  Attention  for  the 
Design  of  Multimodal  Interfaces:  A  Driving  Simulation  Study  In  In  Proceedings  of  50th  Annual  Meeting  of  the 
Human  Factors  and  Ergonomics  Society.  San  Francisco  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  design  of  multimodal  interfaces  rarely  takes  into  consideration  recent  data  suggesting  the  existence  of 
considerable  crossmodal  spatial  and  temporal  links  in  attention.  This  can  be  partly  explained  by  the  fact  that 
crossmodal  links  have  been  studied  almost  exclusively  in  spartan  laboratory  settings  with  simple  cues  and  tasks.  As 
a  result,  it  is  not  clear  whether  they  scale  to  more  complex  settings.  To  examine  this  question,  participants  in  this 
experiment  drove  a  simulated  military  vehicle  and  were  periodically  presented  with  lateralized  visual  indications 
marking  locations  of  roadside  mines  and  safe  areas  of  travel.  Valid  and  invalid  auditory  and  tactile  cues  preceded 
these  indications  at  varying  stimulus-onset  asynchronies.  The  findings  confirm  that  the  location  and  timing  of 
crossmodal  cue  combinations  affect  response  time  and  accuracy  in  complex  domains  as  well.  In  particular, 
presentation  of  crossmodal  cues  at  SOAs  below  500ms  and  tactile  cuing  resulted  in  lower  accuracy  and  longer 
response  times. 


Ferris,  T.  K.,  &  Sarter,  N.  B.  (2008).  Crossmodal  Links  between  Vision,  Audition,  and  Touch  in  Complex 
Environments.  Human  Factors  Journal.  (February  2008). 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Objectives:  This  study  sought  to  determine  whether  performance  effects  of  crossmodal  spatial  links  that  were 
observed  in  earlier  laboratory  studies  scale  to  more  complex  environments  and  need  to  be  considered  in  multimodal 
interface  design.  It  also  revisits  the  unresolved  issue  of  cross-modal  cuing  asymmetries. 

Background:  Previous  laboratory  studies  employing  simple  cues,  tasks,  and/or  targets  have  demonstrated  that  the 
efficiency  of  processing  visual,  auditory,  and  tactile  stimuli  is  affected  by  the  modality,  lateralization,  and  timing  of 
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surrounding  cues.  Very  few  studies  have  investigated  these  cross-modal  constraints  in  the  context  of  more  complex 
environments  to  determine  whether  they  scale  and  how  complexity  affects  the  nature  of  crossmodal  cuing 
asymmetries. 

Method:  Amicroworld  simulation  of  battlefield  operations  with  a  complex  task  set  and  meaningful  visual,  auditory, 
and  tactile  stimuli  was  used  to  investigate  cuing  effects  for  all  cross-modal  pairings. 

Results:  Significant  asymmetric  performance  effects  of  cross-modal  spatial  links  were  observed.  Auditory  cues 
shortened  response  latencies  for  collocated  visual  targets  but  visual  cues  did  not  do  the  same  for  collocated  auditory 
targets.  Responses  to  contralateral  (rather  than  ipsilateral)  targets  were  faster  for  tactually  cued  auditory  targets  and 
each  visual-tactile  cuetarget  combination,  suggesting  an  inhibition-of-return  effect. 

Conclusions:  The  spatial  relationships  between  multimodal  cues  and  targets  significantly  affect  target  response  times 
in  complex  environments.  The  performance  effects  of  cross-modal  links  and  the  observed  cross-modal  cuing 
asymmetries  need  to  be  examined  in  more  detail  and  considered  in  future  interface  design. 

Application:  The  findings  from  this  study  have  implications  for  the  design  of  multimodal  and  adaptive  interfaces 
and  for  supporting  attention  management  in  complex,  data-rich  domains. 


Fiore,  S.  M.,  &  Hoffman,  R.  R.  (2003).  Naturalistic  Decision  Making  and  the  Practice  of  Science  in  National 
Security  Research.  In  In  Proceedings  of  6th  International  Conference  on  Naturalistic  Decision  Making.  Pensacola 
Beach,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Critical  to  meeting  the  needs  arising  from  the  government's  evolving  role  in  our  nation's  security  is  a  broad 
understanding  of  how  to  guide,  and  even  change,  the  interaction  between  science  practice  and  science  policy.  From 
a  practical  standpoint,  it  is  important  that  technologies  be  investigated  for  their  potential  applicability  to  national 
security.  From  a  theoretical  standpoint,  the  concepts  that  drive  the  investigation  and  development  of  new 
technologies  and  the  refinement  of  emerging  technologies  must  also  be  identified.  Because  of  the  tension  that 
inherently  arises  from  dichotomizing  basic  and  applied  research,  we  suggest  that,  via  the  appropriate  blend  of 
ecologically  and  epistemologically  valid  research,  scientists  can  simultaneously  address  these  needs. 

In  this  poster  we  describe  how  Naturalistic  Decision  Making  as  a  paradigm  for  research  provides  a  useful  model  for 
science  practice  associated  with  the  complex  and  dynamic  areas  impacting  national  security.  First  we  elaborate  on 
how  NDM's  reliance  on  both  epistemological  relevance  and  ecological  salience  (see  Hoffman  &  Deffenbacher, 

1993)  articulates  a  model  for  NDM  research  as  it  relates  to  national  security.  Ecological  salience  describes  the 
degree  to  which  the  materials  or  tasks  of  study  pertain  to  what  is  actually  perceived  or  done  in  an  operational  setting. 
Epistemological  relevance  describes  the  degree  to  which  the  experimental  approach  relies  on  concepts  from  extant 
theories.  Via  simultaneous  utilization  of  these  approaches,  we  may  be  able  to  better  develop  an  understanding  of  the 
complexities  of  human  performance  in  national  security  related  tasks.  Second,  we  discuss  examples  of  how 
concepts  derived  from  NDM  can  be  used  as  operationally  relevant  starting  points  for  research  in  national  security. 
These  range  from  understanding  performance  at  the  individual  and  cognitive  level  (e.g.,  expertise  and  perceptual 
processes)  to  the  team  and  organizational  level  (e.g.,  shared  mental  models).  For  example,  research  in  remote 
sensing  imagery  (e.g.,  Hoffman,  Markman,  &  Carnahan,  2001)  and  pattern  recognition  processes  by  experienced 
aviators  (e.g.,  Fiore,  Jentsch,  Oser,  &  Cannon-Bowers,  2000)  are  both  directly  relevant  to  a  number  of  national 
security  areas  (e.g.,  pattern  recognition  by  security  screeners).  In  sum,  we  will  show  how  the  practice  of  science  as 
it  relates  to  research  in  national  security  relevant  areas  can  usefully  apply  the  methods  and  theories  derived  from 
naturalistic  decision  making. 


Fisher,  A.,  McDermott,  P.  L.,  &  Fagan,  S.  (2009).  Bandwidth  Allocation  in  a  Military  Teleoperation  Task.  In  In 
Proceedings  of  Human-Robot  Interaction  Conference,  San  Diego,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Categories  and  Subject  Descriptors 

H.5.2  Information  Interfaces  and  Presentation,  User  Interfaces  - 
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Interaction  styles 
General  Terms 

Human  Factors,  Performance,  Design,  Experimentation 
Keywords 

Human-robot  interaction,  resolution,  frame  rate,  teleoperation, 
color,  grayscale 

Summary  abstract: 

The  implications  of  bandwidth  allocation  are  described  for  teleoperation  in  a  military  task  that  involved  navigation, 
target  detection,  and  target  identification.  Color  versus  grayscale  imagery  was  manipulated.  Participants  themselves 
traded  off  resolution  and  frame  rate  settings.  Participants  minimized  switching  between  resolution/frame  rate 
settings  and  tended  to  use  settings  with  high  resolution/low  frame  rate.  Courses  completed  with  the  highest 
resolution  (and  lowest  frame  rate)  had  the  fastest  target  identification  times,  but  no  other  differences  were  observed 
between  settings.  Color  imagery  offered  advantages  for  overall  course  time  and  the  time  to  identify  a  tank  as 
friendly  or  enemy. 


Flach,  J.  M.,  &  Hoffman,  R.  R.  (2003).  The  Limitations  of  Limitations.  IEEE  Intelligent  Systems,  18(1),  94-96. 

Key  Words: 

ADA  CTA 
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Summary  abstract: 

One  of  the  principles  of  human-centered  computing,  the  Aretha  Franklin  principle,  states,  Do  not  devalue  the  human 
in  order  to  justify  the  machine.  Do  not  criticize  the  machine  in  order  to  rationalize  the  human.  Advocate  the  human- 
machine  system  in  order  to  amplify  both.  The  implication  we  pursue  in  this  article  has  to  do  with  the  notion  of 
human  limitations.  In  the  introduction  to  his  text  on  engineering  psychology,  Christopher  Wickens  wrote,  “One 
major  purpose  of  this  book  is  to  examine  human  capabilities  and  limitations  in  the  specific  area  of  information 
processing.  The  second  purpose  is  to  demonstrate  how  knowledge  of  these  limitations  can  be  applied  in  the  design 
of  complex  systems  with  which  humans  interact. ”2  The  first  sentence  in  this  quotation  clearly  recognizes  that 
humans  have  capabilities.  However,  the  second  sentence  seems  to  imply  that  it  is  the  limitations  that  are  most 
relevant  in  the  design  of  complex  systems.  In  a  similar  vein,  Barry  Kantowitz  and  Robert  Sorkin  wrote,  “Indeed, 
many  human  factors  analysts  believe  that  minimizing  human  error  is  the  primary  goal  of  any  human  factors  design. 
If  people  never  made  errors,  there  would  be  little  need  for  a  science  of  human  factors”  (italics  added). 3  Such 
statements,  and  there  are  scads  of  them  in  the  literature,  portray  the  human  as  the  weaker  link  in  any  complex 
system.  And  the  design  focus  tends  to  be  on  protecting  the  system  from  the  limitations  and  errors  that  are  associated 
with  that  weak  link.  In  this  essay,  we  argue  that  human  factors  and  applied  cognitive  psychologists  have  not  just 
been  selective  in  regarding  certain  human  characteristics  as  limitations,  but  also  have  selected  the  wrong  things  and 
for  the  wrong  reasons. 
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M.  Gernsbacher,  &  S.  Goldman  (Eds.),  The  Handbook  of  Discourse  Processes.  Mahwah,  NJ:  Lawrence  Erlbaum 
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For  the  field  of  discourse  processing  to  advance,  researchers  must  understand  the  underlying  cognitive  processes 
involved.  The  creation  of  models  of  discourse  processing  permit  researchers  to  develop  predictions  based  on  their 
understanding  of  these  cognitive  processes.  For  these  models  to  be  testable,  they  must  be  implemented  in  a  way  that 
they  can  generate  specific  quantitative  predictions  that  can  be  compared  against  research  data.  Current  quantitative 
models  have  proved  to  be  successful  in  accomplishing  these  goals.  Researchers  can  run  simulations  of  human 
discourse  processing  and  test  the  effectiveness  of  the  models.  This  permits  an  improved  understanding  of  how 
humans  process  discourse,  as  well  as  the  development  of  practical  applications  that  simulate  human  processing. 


Foltz,  P.  W.  (2003).  Book  Chapter:  Content  Processing  with  Latent  Semantic  Analysis:  Essays,  Teams  and  Clinical 
Interviews.  In  L.  Lagerwerf,  W.  Spooren,  &  L.  Degand  (Eds.),  Determination  of  Information  and  Tenor  in  Texts: 
Multidisciplinary  Approaches  to  Discourse.  Munster,  Germany:  Nodus  Publikationen. 
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Summary  abstract: 

Latent  Semantic  Analysis  (LSA)  is  a  computational  model  of  human  knowledge  acquisition  and  a  practical 
application  for  information  retrieval  and  concept-based  text  matching.  This  paper  illustrates  w~ys  in  which  LSA  can 
be  applied  to  performing  content  analyses  of  text  and  discourse.  Three  example  domains  are  shown:  The  analysis  of 
student  essays  forjudging  knowledge,  the  analysis  ofteam  discourse  for  predicting  team  performance  and  automatic 
tagging  of  discourse  content,  and  the  analysis  of  clinical  interviews  of  patients  with  schizophrenia  for  predicting  the 
presence  and  severity  of  the  disease.  The  results'  show  that  overall,  LSA  can  be  used  in  these  domains  for  modeling 
cognitive  and  social  properties  as  well  as  providing  useful  applications  that  make  predictions  based  on  automated 
discourse  analysis. 


Foltz,  P.  W.,  Laham,  D.,  &  Derr,  M.  (2003).  Automated  Speech  Recognition  for  Modeling  Team  Performance.  In  In 
Proceedings  of  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 
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Summary  abstract: 

While  team  tasks  provide  a  wealth  of  data  on  individual  and  team  performance,  techniques  for  modeling  team 
communication  can  be  quite  effortful  and  time-consuming.  Automated  techniques  of  analyzing  team  discourse 
provide  the  promise  of  quickly  judging  team  performance  and  permitting  feedback  to  teams  both  in  training  and  in 
operations.  In  previous  research,  techniques  using  Latent  Semantic  Analysis  (LSA)  have  proven  successful  for 
analyzing  team  transcripts.  However,  converting  the  audio  discourse  into  transcripts  often  requires  hand 
transcription.  In  this  work,  we  describe  applying  automated  speech  recognition  (ASR)  to  team  transcripts  and  using 
the  output  of  the  ASR  to  predict  overall  team  performance.  Results  indicate  that  ASR  can  be  used  in  conjunction 
with  semantic  methods  of  modeling  team  communication  to  provide  accurate  predictions  of  performance.  The  work 
has  potential  for  assisting  operators  in  the  performance  of  their  tasks  because  it  can  “listen”’  and  in  real-time 
evaluate  fieform  verbal  communication  from  a  variety  of  sources. 


Foltz,  P.  W.,  &  Lee,  A.  Y.  (2004).  Chapter  9.  Adaptive  Learning  Systems:  Toward  More  Intelligent  Analyses  of 
Student  Responses.  In  V.  Oostendorp,  Breure,  &  Dillon  (Eds.),  Creation,  Use,  and  Deployment  of  Digital 
Information:  NMSU. 
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Summary  abstract: 

A  critical  issue  for  using  information  systems  is  the  need  for  assessing  what  people  have  learned  and  for  using  that 
information  to  provide  substantive  additional  instruction.  The  creation  of  electronic  training  material  is  relatively 
easy;  however,  creating  systems  that  truly  adapt  to  an  individual  student  is  more  difficult.  For  example,  textbooks 
can  be  scanned  to  create  electronic  textbooks  for  students  to  read,  but  these  electronic  textbooks  may  be  no  better 
than  the  paper  version  if  they  do  not  assess  what  the  student  knows  or  present  only  material  the  student  needs. 


Foltz,  P.  W.,  Lee,  A.  Y.,  Bond,  G.,  &  Martin,  M.  (2009).  Chapter  22.  Models,  Methods,  and  Tools  for  the 
Adaptability  of  Teams.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the 
Warfighter:  Foundations  and  Technology.  Section  IV.  Communicating  Information  across  the  Team:  48hrBooks 
by  Alion  Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen 
Proving  Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced 
Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management 
Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1100  New  Jersey 
Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 
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Summary  abstract: 

SSTR  (Stability,  Security,  Transition  and  Reconstruction)  missions  often  require  that  diverse,  distributed  experts 
from  multinational  forces,  non-governmental  organizations,  and  other  government  agencies  work  together. 
Commanders  and  teams  trained  as  Warfighters  often  must  acquire  different  skills  to  perform  effectively  for  STIR 
operations.  While  some  knowledge  can  transferfrom  warfighting  skills,  other  skills  may  be  needed,  such  as  the 
ability  to  perform  effective  decision-making  in  novel  cultural  situations  and  developing  multi-functional  teams  from 
coalition  forces.  This  raises  a  number  of  issues  about  how  to  effectively  assess  and  measure  performance  in  such 
operations  and  how  tools  can  be  developed  that  provide  appropriate  support  during  SSTR  operations. 

The  domain  of  SSTR  operations  provides  a  rich  area  for  examining  cognitive,  social,  and  cultural  aspects  of 
collaboration  while  also  providing  information  that  should  be  generalizable  to  military  team  performance  over  a 
wide  number  of  different  domains.  For  example,  commanders  and  team  members  must  understand  how  cultural  and 
social  differences  influence  behaviors  such  as  information  sharing  and  managing  uncertainty  or  risk.  However, 
collaborative  technology  that  merely  supports  information  exchange  will  be  necessary,  but  not  sufficient  to  insure 
collaboration  among  highly  diverse  team  members  solving  complex  problems.  The  overall  objective  of  the  present 
work  is  to  develop  and  validate  models,  methods,  and  automated  tools  for  measurement  of  performance  in  teams 
performing  Stability,  Security,  Transition,  and  Reconstruction  (SSTR)  operations. 

The  work  described  in  this  chapter  focuses  on  methods  to  assess  and  enhance  collaboration  in  teams  and  the  effects 
of  culture  and  social  issues  (e.g.,  trust)  on  team  performance.  Learning  and  transfer  of  the  teams'  skills  was 
examined  by  having  teams  perform  in  an  SSTR  task  environment  where  they  were  either  collocated  with  other  team 
members  or  separated  and  only  able  to  communicate  through  collaborative  systems.  Finally,  automated  methods 
were  developed  to  predict  team  performance  through  analysis  of  the  communication  among  the  team  members.  We 
first  describe  some  ofthe  issues  associated  with  the  areas  of  learning  and  transfer,  trust  in  teams,  and  automated 
communication  analysis.  Then  we  provide  an  overview  of  experiments  performed  to  examine  these  issues. 


Foltz,  P.  W.,  Martin,  M.  J.,  Cooke,  N.  J.,  Kiekel,  P.  A.,  &  Gorman,  J.  (2003).  Automated  Team  Communication 
Analysis  with  Latent  Semantic  Analysis.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003. 
Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland 
University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 
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Summary  abstract: 

While  team  tasks  provide  a  wealth  of  data  on  individual  and  team  performance,  techniques  for  modeling  team  tasks 
can  be  quite  effortful  and  time-consuming.  Automated  techniques  of  team  modeling  provide  the  promise  of  quickly 
judging  team  performance  and  permitting  feedback  to  teams  both  in  training  and  in  operations.  One  of  the  most 
complex  aspects  of  modeling  team  performance  is  modeling  team  member's  dialogues.  We  describe  the  use  of 
Latent  Semantic  Analysis  (LSA)  for  automatically  analyzing  and  coding  team  discourse.  LSA  is  a  fully  automatic 
corpus-based  statistical  method  for  extracting  and  nferring  relations  of  expected  contextual  usage  of  words  in 
discourse.  In  this  research,  we  describe  two  approaches  to  analyzing  team  discourse.  The  first  approach  measures  the 
semantic  content  of  the  team  dialogue  as  a  whole  and  accurately  predicts  the  team's  performance  on  a  simulated 
military  mission.  The  second  approach  categorizes  the  contents  of  each  statement  made  by  team  members  using  an 
established  set  of  tags.  These  tagged  statements  can  then  be  used  to  predict  team  performance.  Performance  on 
tagging  was  only  15%  below  the  performance  of  trained  human  annotators.  Future  improvements,  and  implications 
for  the  design  of  automated  systems  for  measuring  team  communications  are  discussed. 


French,  J.,  Ghirardelli,  T.  G.,  Swoboda,  J.,  Flo,  S.,  Nguyen,  H.,  Tokarcik,  L.,  Walrath,  J.,  &  Winkler,  R.  (2003). 
Modes  of  Control  of  an  Unmanned  Ground  Vehicle  (UGV).  In  In  Proceedings  of  Collaborative  Technology 
Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003. 
University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

Unmanned  Ground  Vehicles  (UGVs)  are  envisioned  for  a  variety  of  functions  that  can  provide  increased  situation 
awareness  (SA)  and  understanding  to  commanders  as  well  as  to  the  Objective  Force  Warrior.  The  objective  of  this 
study  was  to  measure  the  effects  of  different  modes  of  control  (joystick,  voice,  joystick  and  voice  in  combination, 
and  simulated  autonomous  control)  of  a  UGV  on  operator  performance.  We  also  examined  simulated  degradation  of 
video  signal  strength.  Participants  were  22  soldiers  recruited  from  the  16th  Ordnance  Battalion  at  Aberdeen  Proving 
Ground,  MD.  Each  was  randomly  assigned  to  one  of  the  four  control  modes  between-group  conditions  and 
experienced  all  four  levels  of  the  video  quality  conditions.  We  found  differences  in  navigation  performance  and 
workload  measures. 


Galie,  J.,  Ho,  H.-N.,  &  Jones,  L.  A.  (2009).  Influence  of  Contact  Conditions  on  Thermal  Responses  of  the  Hand.  In 
In  Proceedings  of  Third  Joint  Eurohaptics  Conference  and  Symposium  on  Haptic  Interfaces  for  Virtual  Environment 
and  Teleoperator  Systems  Salt  Lake  City,  UT,  USA,  March  18-20,  2009  (pp.  587-592). 
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Keywords:  temperature,  thermal  display,  perception 
Summary  abstract: 

A  series  of  experiments  was  conducted  to  evaluate  how  contact  pressure  and  surface  roughness  influence  the  heat 
flux  conducted  out  of  the  skin  or  object  during  contact.  Changes  in  skin  temperature  assist  in  identifying  objects  held 
in  the  hand.  In  the  first  experiment  an  infrared  thermal  imaging  system  was  used  to  measure  skin  temperature  and 
contact  area  as  participants  generated  forces  ranging  from  0.1  to  6  N  with  their  index  finger.  The  results  showed  that 
skin  temperature  decreased  by  an  average  of  5.5  oC  across  the  range  of  forces  studied  and  that  the  changes  were 
greatest  between  0.25-0.35  N  and  from  4-6  N.  The  second  and  third  experiments  examined  the  effect  of  the  surface 
roughness  of  an  object  on  skin  temperature  and  on  the  perceived  coldness  of  the  object.  A  set  of  six  copper  blocks 
was  machined  to  create  a  range  of  surface  profiles.  There  was  a  slight  decrease  in  skin  temperature  as  the  surface 
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roughness  of  the  object  increased,  contrary  to  theoretical  predictions.  Although  small,  these  changes  were 
perceptible  as  participants  consistently  chose  the  rougher  of  two  stimuli  when  asked  to  select  the  cooler  stimulus. 
These  results  indicate  that  contact  pressure  and  surface  roughness  influence  the  change  in  skin  temperature  during 
contact  and  that  they  can  have  a  perceptible  influence  on  the  perceived  properties  of  objects  held  in  the  hand. 
Thermal  models  need  to  account  for  these  effects  if  realistic  feedback  is  to  be  presented  in  a  thermal  display. 


Gillan,  D.  J.  (2008).  Human-Robot  Interaction.il:  The  Impact  of  Human  Factors  on  Unmanned  Systems.  In  In 
Proceedings  of  52nd  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  22-26  September  2008.  New 
York  City,  NY  USA. 
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Summary  abstract: 

The  sophistication  of  robotics  systems  has  increased  the  utility  of  unmanned  systems  for  complex  military 
operations.  The  increasing  capability  and  functionality  of  current  and  future  robotics  requires  comparable 
enhancements  to  robotics  interfaces  and  displays  for  tasking  and  controlling  unmanned  systems.  The  symposium 
participants  identified  key  issues  and  challenges  in  human-robot  interaction  with  respect  to  situation  awareness 
(SA),  perception  and  performance,  and  workload.  The  purpose  of  the  symposium  is  to  discuss  current  research 
findings  and  how  these  findings  can  be  applied.  The  authors  and  audience  will  discuss  the  challenges  of  conducting 
human-robot  interaction  (HRI)  research  and  the  lessons  learned  for  the  design  of  human-robotic  interactions  and 
interfaces. 


Gillan,  D.  J.,  &  Gillan,  C.  T.  (2006).  Effects  of  Motion  on  the  Perception  of  Static  Features  in  a  Display.  In  In 
Proceedings  of  50th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  San  Francisco  CA. 
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Summary  abstract: 

Displays  that  use  motion  typically  contain  both  dynamic  and  static  features.  We  hypothesized  that,  relative  to  a  no 
motion  condition,  representational  momentum  (the  forward  displacement  of  the  representation  of  a  moving  object) 
might  influence  the  perception  of  static  distance  between  two  objects  when  the  objects  had  just  moved  away  from 
each  other  or  just  moved  toward  each  other.  Two  experiments  tested  the  hypothesis  by  manipulating  the  end 
distances  between  two  moving  dots  and  the  type  of  motion  (Motion  Away,  Motion  Towards,  No  Motion).  In 
Experiment  1,  the  distance  that  the  dots  moved  was  the  same  for  all  six  end  distances,  whereas  in  Experiment  2,  the 
distance  moved  was  proportional  to  the  end  distance.  Participants  estimated  the  distance  at  the  end  of  the  motion  on 
every  trial.  The  results  of  both  experiments  showed  that  the  Motion  Away  produced  greater  estimates  of  distance 
than  the  other  conditions.  This  effect  was  most  pronounced  at  the  longest  end  distances  in  Experiment  2.  The  results 
suggest  designers  should  be  cautious  in  using  motion  in  displays  for  tasks  that  involve  estimates  of  static  distance. 
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Technology  extends  human  capabilities.  For  much  of  human  history,  technological  development  focused  on 
enhancing  physical  abilities  —  for  example,  hammers  increase  our  strength;  transportation  technologies  extend  our 
ability  to  move  (Gillan  &  Schvaneveldt,  1999).  In  contrast,  many  technological  innovations  since  the  mid-20th 
Century  have  focused  on  enhancing  human  cognitive  abilities  -  for  example,  computing  technologies  extend  our 
ability  to  process,  store,  and  represent  symbolic  information  (Gillan  &  Schvaneveldt,  1999). 

When  the  Czech  playwright,  Karel  Capek,  first  coined  the  term  robot  (from  the  Slav,  robota,  meaning  work  or 
forced  labor)  in  his  1921  play,  Rossum’s  Universal  Robots,  he  foresaw  technological  development  that  would 
extend  the  ability  to  perform  virtually  any  task  (Capek,  1921/2004).  Interestingly,  he  foresaw  the  use  of  robots  to 
perform  both  physical  and  intellectual  tasks  since  the  Czech  word,  rozum,  means  “mind”  or  “intellect”.  The  robots 
that  Capek  described  were  human-like  androids  capable  of  supplanting  humans.  In  the  play,  the  robots  seem  to 
value  physical  work  in  that  they  spare  the  life  of  one  human,  based  on  their  observation  that  he  works  with  his 
hands. 

As  in  Capek’s  vision  of  robots,  modern  robots’  capabilities  span  a  wide  variety  of  tasks.  We  constrain  this  chapter 
by  focusing  on  a  set  of  robotic  tasks  related  to  military  contexts.  The  goal  of  this  chapter  is  to  examine  selected 
cognitive  processes  that  operators  of  robotic  systems  will  need  to  apply  as  they  control,  supervise,  and/or  monitor 
robots  performing  these  tasks.  Consequently,  we  have  also  restricted  our  analysis  of  cognitive  processes  to  those 
most  closely  associated  with  an  operator’s  activities  while  a  robot  performs  one  of  these  tasks. 


Gillan,  D.  J.,  &  Sapp,  M.  V.  (2005).  Static  Representation  of  Object  Motion.  In  In  Proceedings  of  49th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 
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Summary  abstract: 

Displaying  motion  information  is  useful  in  dynamic  tasks,  such  as  tracking  or  predicting  the  course  of  moving 
objects  or  systems  (e.g.,  unmanned  ground  or  air  vehicles,  troops,  or  weather).  Static  representations  of  object 
motion  may  be  useful  when  technological  limitations  prevent  use  of  dynamic  displays.  The  present  experiment 
examined  people’s  interpretations  of  a  variety  of  static  cues  to  represent  object  motion.  Participants  viewed  and 
rated  two  static  types  of  representation  of  object  motion  —  motion  lines  and  arrows.  The  features  of  object  motion 
that  participants  rated  were  distance  traveled,  direction,  path,  speed,  and  acceleration.  The  results  show  that 
observers  reliably  interpret  certain  static  cues  to  represent  features  of  object  motion,  especially  distance,  direction, 
path  and  speed;  the  cues  examined  were  not  interpreted  as  representing  acceleration.  The  results  are  interpreted  as 
guidelines  for  design  of  displays  that  include  object  motion  information. 


Gonzalez,  C.  (2002).  The  Role  of  Cognitive  Modeling  in  Enhancing  Dynamic  Decisions.  In  In  Proceedings  of  24th 
Annual  Meeting  of  the  Cognitive  Science  Society.  Fairfax,  VA.  (pp.  1005). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Dynamic  Decision  Making  is  characterized  by  multiple  and  interdependent  decisions,  autonomous  environments, 
and  real-time  evaluation  and  action  (Brehmer,  1990).  There  seems  to  be  increasing  agreement  on.  how  we  make 
decisions  in  this  type  of  situations.  Decision  makers  in  dynamic  environments  recognize  typical'  situations  and 
typical  responses  and  use  their  knowledge  to  adapt  their  strategies  "on  the  fly"  (payne,  Bettman,  and  Johnson,  1993; 
Klein,  1998). 


Gonzalez,  C.  (2003).  ACT-R  Implementation  of  an  Instance-Based  Decision  Making  Theory.  In  In  Proceedings  of 
Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective 
Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research 
Laboratory. 
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Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

The  Instance-Based  Learning  Theory  (IBLT)  proposes  that  in  dynamic  decision-making  (DDM)  decision  makers 
achieve  control  over  a  system  through  recognition,  accumulation  and  refinement  of  decision-making  instances.  Each 
instance  contains  information  on  a  situation,  an  action,  and  a  utility  of  a  decision.  IBLT  is  a  general  theory  that 
explains  how  people  make  decisions  in  dynamic  environments  and  CogIBLT  is  the  implementation  of  such  theory  in 
a  cognitive  architecture,  ACT-R.  This  paper  introduces  IBLT,  describes  its  implementation  in  ACT-R,  its 
predictions,  and  discusses  the  implications  of  this  work  to  Objective  Force. 


Gonzalez,  C.,  Lerch,  J.  F.,  &  Lebiere,  C.  (2003).  Instance-Based  Learning  in  Dynamic  Decision  Making.  Cognitive 
Science,  27(4),  591-635. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Dynamic  decision  making;  Instance -based  learning;  Cognitive  modeling;  Decision  making; 

Water  purification  plant 

Summary  abstract: 

This  paper  presents  a  learning  theory  pertinent  to  dynamic  decision  making  (DDM)  called  instancebased  learning 
theory  (IBLT).  IBLT  proposes  five  learning  mechanisms  in  the  context  of  a  decision-making  process:  instance -based 
knowledge,  recognition-based  retrieval,  adaptive  strategies,  necessity-based  choice,  and  feedback  updates.  IBLT 
suggests  in  DDM  people  learn  with  the  accumulation  and  refinement  of  instances,  containing  the  decision-making 
situation,  action,  and  utility  of  decisions.  As  decision  makers  interact  with  a  dynamic  task,  they  recognize  a  situation 
according  to  its  similarity  to  past  instances,  adapt  their  judgment  strategies  from  heuristic-based  to  instance -based, 
and  refine  the  accumulated  knowledge  according  to  feedback  on  the  result  of  their  actions.  The  IBLT’s  learning 
mechanisms  have  been  implemented  in  an  ACT-R  cognitive  model.  Through  a  series  of  experiments,  this  paper 
shows  how  the  IBLT’s  learning  mechanisms  closely  approximate  the  relative  trend  magnitude  and  performance  of 
human  data.  Although  the  cognitive  model  is  bounded  within  the  context  of  a  dynamic  task,  the  IBLT  is  a  general 
theory  of  decision  making  applicable  to  other  dynamic  environments. 


Gonzalez,  C.  (2003).  Verbal  Protocols  in  Real-Time  Dynamic  Decision-Making  In  In  Proceedings  of  47th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  study  presents  the  results  from  the  analyses  of  verbal  protocols  elicited  from  inexperienced  and  experienced 
participants  of  a  real-time,  Dynamic  Decision-Making  (DDM)  task.  This  research  intends  to  complement  a  series  of 
studies  performed  in  DDM  environments  analyzing  the  cognitive  structures  and  processes  involved  in  learning  in 
DDM.  Results  show  that  inexperienced  and  experienced  participants  differ  in  several  ways:  in  the  way  they 
distribute  attention  to  different  parts  of  the  system,  in  their  awareness  of  the  relationship  of  the  attributes  involved  in 
the  decision  making  process,  and  in  their  coordination  to  make  decisions  in  real  time.  These  results  have  been  used 
to  support  the  refinement  of  a  cognitive  model  developed  to  explain  how  people  learn  in  DDM  tasks. 


Gonzalez,  C.  (2004).  The  Relationships  between  Cognitive  Ability  and  Performance  in  Dynamic  Decision  Making. 
Intelligence  Journal,  33(2). 
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Journal  Article 

Cognitive  ability;  Dynamic  decision  making;  Practice-Centered  Design. 

Summary  abstract: 

This  study  investigated  the  relationships  between  cognitive  ability  (as  assessed  by  the  Raven  Progressive  Matrices 
Test  [RPM]  and  the  Visual-Span  Test  [VSPAN])  and  individuals’  performance  in  three  dynamic  decision  making 
(DDM)  tasks  (i.e.,  regular  Water  Purification  Plant  [WPP],  Team  WPP,  and  Firechief).  Participants  interacted 
repeatedly  with  one  of  the  three  microworlds.  Our  results  indicate  a  positive  association  between  VSPAN  and  RPM 
scores  and  between  each  of  those  measures  and  performance  in  the  three  dynamic  tasks.  Practice  had  no  effect  on 
the  correlation  between  RPM  score  and  performance  in  any  of  the  microworlds,  but  it  led  to  an  increased  correlation 
between  VSPAN  score  and  performance  in  Team  WPP.  The  pattern  of  associations  between  performance  in 
microworlds  and  assessments  of  cognitive  ability  was  consistent  with  the  task  requirements  of  the  microworlds. 
These  findings  provide  insight  into  the  cognitive  demands  of  dynamic  decision  making  and  the  dynamics  of  the 
relationships  between  cognitive  ability  and  performance  with  task  practice. 


Gonzalez,  C.  (2005).  Decision  Support  for  Real-Time  Dynamic  Decision  Making  Tasks.  Organizational  Behavior 
and  Human  Decision  Processes  Journal,  96,  142-154. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Dynamic  decision  making;  Cognitive  support;  Feedback;  Feedforward;  Cognitive  feedback 
Summary  abstract: 

By  definition,  dynamic  decision  making  dictates  that  multiple  and  interrelated  decisions  be  made  in  a  continuously 
changing  environment.  Such  decision  making  is  difficult  and  often  taxes  individuals’  cognitive  resources.  Here  I 
investigated  ways  in  which  to  support  decision  making  in  these  environments.  I  evaluated  three  forms  of  decision 
support:  outcome  feedback,  cognitive  feedback,  and  feedforward  that  incorporated  (to  varying  degrees)  common 
features  of  learning  theories  associated  with  dynamic  tasks.  Participants  in  a  laboratory  experiment  performed  a  real¬ 
time,  dynamic  decision-making  task  while  receiving  one  of  the  diVerent  types  of  decision  support.  During  the  Wrst 
2  days,  individuals  received  one  type  of  decision  support,  but  on  the  third  day  they  performed  the  task  without  this 
support.  Participants  who  received  feedforward  improved  their  performance  considerably  and  continued  to  exhibit 
improved  performance  even  after  discontinuation  of  the  decision  support  on  the  third  day.  Neither  outcome  feedback 
nor  cognitive  feedback  resulted  in  improved  performance.  More  research  is  necessary  to  conclusively  identify  the 
best  forms  of  dynamic  decisionmaking  support  and  their  durability  when  transferred  to  new  tasks. 


Gonzalez,  C.  (2005).  Learning  to  Make  Decisions  in  Dynamic  Enrivonments:  Effects  of  Time  Constraints  and 
Cognitive  Abilities.  Human  Factors,  46(3),  449-460. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

This  study  investigated  the  effects  of  time  constraints  and  cognitive  abilities  on  dynamic  decision  making  (DDM). 
The  learning  and  performance  of  individuals  trained  in  a  DDM  task  with  time  constraints  were  compared  with  those 
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who  were  trained  without  time  constraints.  Although  all  participants  received  the  same  total  amount  of  time  to 
perform  the  task,  individuals  under  more  stringent  time  constraints  were  given  3  times  more  practice  trials  on  the 
first  2  days  of  the  study  than  were  people  under  less  stringent  time  constraints.  Despite  the  additional  practice  runs, 
participants  under  high  time  constraints  performed  worse  than  did  participants  under  low  time  constraints  on  the  3rd 
day  of  the  study.  A  subsequent  analysis  of  cognitive  abilities  and  decision  heuristics  revealed  that  individuals’ 
actions  corresponded  with  simple  heuristic  predictions  more  closely  with  minimal  practice  than  with  extended 
practice,  under  high  rather  than  low  time  constraints,  and  in  individuals  with  low  rather  than  high  cognitive  abilities. 
These  findings  suggest  that  mere  repetition  of  a  task  with  less  time  within  a  trial  is  counterproductive  to  learning  and 
that  learning  depends  on  cognitive  abilities.  Potential  applications  of  this  research  include  the  design  of  training 
procedures  for  dynamic  tasks. 


Gonzalez,  C.  (2005).  Task  Workload  and  Cognitive  Abilities  in  Dynamic  Decision  Making.  Human  Factors,  47(1), 
92-101. 

Key  Words: 
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Summary  abstract: 

Researchers  often  treat  workload  as  a  task-dependent  variable.  To  evaluate  the  effect  of  workload  on  individuals’ 
performance,  researchers  commonly  use  several  methods,  such  as  varying  the  complexity  or  number  of  tasks  that 
test  participants  are  asked  to  handle  or  placing  individuals  under  time  constraints.  Only  rarely  have  researchers 
investigated  workload  as  a  variable  dependent  on  individuals’  cognitive  abilities.  This  study  investigated  workload 
during  dynamic  decision  making  in  terms  of  its  dependence  on  both  task  workload  and  cognitive  abilities.  The 
findings  demonstrate  detrimental  effects  of  both  high  task  workload  and  low  cognitive  abilities.  Further,  the  results 
show  that  high  workload  is  more  detrimental  in  individuals  with  low  cognitive  abilities  than  in  individuals  with  high 
cognitive  abilities.  Potential  applications  of  this  research  include  the  design  of  new  workload  studies  and  new 
training  protocols  in  which  psychometric  tests  are  used. 


Gonzalez,  C.,  &  Golenbock,  J.  (2003).  Impact  of  Numerical  and  Graphical  Formats  on  Dynamic  Decision  Making 
Performance:  An  Eye-Tracking  Study.  In  In  Proceedings  of  First  Latin  American  Conference  of  Human-Computer 
Interaction.  Rio  de  Janeiro,  Brazil. 

Key  Words: 
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Summary  abstract: 

This  paper  presents  a  study  in  which  we  manipulated  the  interface  of  a  computer  simulation  into:  graphical  and 
numerical  formats.  We  obtained  both  performance  and  eye-tracking  learning  curves  from  individuals  assigned  to 
one  of  these  two  conditions.  Our  findings  indicate  that  although  performance  is  not  different  between  the  two 
interfaces,  the  amount  of  attention  as  measured  by  the  number  of  eye-tracking  points  was  very  different  in  the 
graphical  and  numerical  conditions.  Attention  increased  over  time  in  the  numerical  condition,  but  was  stable  in  the 
graphical  condition.  These  results  showed  that  the  strategies  used  to  make  decisions  in  dynamic  environments  vary 
according  to  the  form  of  information  presentation. 


Gonzalez,  C.,  Juarez,  O.,  &  Giloni,  J.  (2002).  Modeling  Coordination  in  Team  Dynamic  Resource  Allocation.  In  In 
Proceedings  of  Sixth  Annual  ACT-R  Workshop.  Carnegie  Mellon  University.  Pittsburgh,  PA. 
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Summary  abstract: 

Dynamic  Decision  Making  (DDM) 

-Multiple  and  interdependent  decisions 

-Continuously  changing  independent  environment  (exogenous  events) 

-Decisions  made  in  real-time 
Coordination  in  Team  DDM 
-Coordination 

^“Managing  dependencies  between  activities”  Malone  and  Crowston  (1994) 

=  “Joint  efforts  of  independent  actors  towards  mutually  defined  goals”  [NSF,  1989] 

-  Coordination  in  Team  DDM 

=  Allocation  of  tasks  and  sharing  information  should  occur  at  the  right  time  and  accurately 

-  How  is  coordination  developed  in  a  team  dynamic  task? 


Gonzalez,  C.,  Martin,  M.,  &  Hansberger,  J.  T.  (2006).  Feedforward  Effects  on  Predictions  in  a  Dynamic  Battle 
Scenario.  In  In  Proceedings  of  50th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  San  Francisco 
CA.  (pp.  265-269). 

Key  Words: 
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Summary  abstract: 

Commanders  face  many  challenges  in  their  efforts  to  control  the  battlefield.  Friction  (i.e.,  sources  of  delay)  in  the 
commander’s  control  system,  coupled  with  the  dynamics  of  the  battlefield,  requires  commanders  to  act  before 
threatening  battlefield  events  occur.  Effective  control  of  the  battlefield  thus  requires  accurate  predictions.  This  paper 
describes  the  results  of  a  preliminary  study  concerned  with  the  effect  of  FeedForward  (FF)  on  the  accuracy  of 
predictions  in  dynamic  battle  situations.  FF,  given  in  the  form  of  expert  advice  prior  to  simulated  battle,  did  not 
reliably  improve  predictions.  Exploratory  analyses,  however,  indicate  that  FF  guided  attention  to  a  subset  of  the  task 
variables  important  for  accurate  prediction.  Furthermore,  FF  produced  quicker  and  more  decisive  victories  than 
practice  alone.  In  conjunction  with  the  positive  performance  trend  for  the  FF  group,  these  findings  indicate  that  FF 
facilitates  strategy  development  and  may  lead  to  higher  levels  of  Dynamic  Decision  Making  (DDM)  performance 
over  time. 


Gonzalez,  C.,  &  Quesada,  J.  (2004).  Learning  in  Dynamic  Decision  Making:  The  Recognition  Process. 
Computational  &  Mathematical  Organization  Theory,  9(4),  287-304. 
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cognitive  representation,  dynamic  decision  making,  recognition 
Summary  abstract: 

The  apparent  difficulty  that  humans  experience  when  asked  to  manage  dynamic  complexity  might  be  related  to  their 
inability  to  discriminate  among  familiar  classes  of  objects  (i.e.,  flawed  recognition).  In  this  study  we  examined  the 
change  in  individuals’  recognition  ability,  as  measured  by  the  change  in  the  similarity  of  decisions  they  made  when 
confronted  repeatedly  with  consistent  dynamic  situations  of  varying  degrees  of  similarity.  The  study  generated  two 
primary  findings.  First,  decisions  became  increasingly  similar  with  task  practice,  a  result  that  suggests  gradually 
improving  discrimination  by  the  participants.  Second,  the  similarity  was  determined  by  the  interaction  of  many  task 
features  rather  than  individual  task  features.  The  general  principles  highlighted  by  this  study  are  applicable  to 
dynamic  situations.  For  example,  with  practice,  decision  makers  should  be  able  to  learn  to  identify  the  time  at  which 
to  intervene  to  achieve  the  maximal  effect  during  dynamic  decision  making. 
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Gonzalez,  C.,  Saner,  L.,  Endsley,  M.,  Bolstad,  C.  A.,  &  Cuevas,  H.  M.  (2009).  Chapter  14.  Modeling  and 
Measuring  Situation  Awareness  in  Individuals  and  Teams.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced 
Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  III.  Acting  on  Battlefield 
Information:  48hrBooks  by  Alion  Science  &  Technology  Coiporation  under  contract  to  the  U.S.  Army  Research 
Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings 
from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final 
Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23, 
2009.  1100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr- 
webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

Situation  Awareness  (SA)  is  a  complex  construct  that  cannot  be  fully  understood  from  a  single  perspective.  Rather, 
SA  entails  a  multifaceted  process  in  which  individual  and  team  factors,  at  both  the  micro  and  macro  levels,  need  to 
be  integrated.  For  example,  research  on  team  SA  needs  to  take  into  account  the  factors  that  contribute  to  a  given 
person's  individual  SA  while  integrating  the  factors  that  contribute  to  any  two  team  members'  shared  SA,  as  well  as 
team  or  organizational  SA.  Also,  the  complexity  of  SA  arises  from  multiple  individual  cognitive  abilities  including 
learning,  working  memory,  different  levels  of  perception,  understanding,  projection,  etc.  These  individual  factors 
interact  with  multiple  team  and  organizational  factors  such  as  geographical  distribution,  collaborative  tool  usage, 
network  proximity,  similarity  of  the  individuals'  background  experiences,  and  familiarity  with  each  others'  skills, 
among  others. 

In  this  chapter  we  bring  together  different  levels  and  factors  of  SA  complexity  into  an  integrated  framework. 
Throughout  the  duration  of  the  Army  Research  Laboratories  Advanced  Decisions  Architectures  (ARLADA) 
research  program,  we  have  addressed  the  complexity  of  SA  by  advancing  the  development  arid  validity  of  measures 
at  the  individual  and  team  levels.  We  have  also  created  computational  models  of  SA  that  have  moved  forward  the 
descriptive  nature  of  SA  into  more  concrete  and  formal  representations.  This  chapter  summarizes  all  the  past  work 
we  have  done  within  this  program  and  attempts  to  integrate  all  the  previous  findings  into  a  research  framework.  The 
details  of  each  of  these  research  pieces  can  be  found  in  the  previous  original  publications.  The  integrated  framework 
presented  here  came  from  a  combination  of  scientific  research  methods  at  the  micro  and  macro  levels  that  included 
computational  models,  and  laboratory  and  field  studies.  The  framework  integrates  our  past  findings  from 
experimental  laboratory  studies  and  ACT-R  cognitive  computational  models  on  individual  SA  to  field  experiments 
and  social  network  analysis  of  team  and  shared  SA. 


Gonzalez,  C.,  Vanyukov,  P.,  &  Martin,  M.  K.  (2005).  The  Use  of  Microworlds  to  Study  Dynamic  Decision  Making. 
Computers  in  Human  Behavior,  21(2),  273-286. 
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Summary  abstract: 

Dynamic  decision-making  (DDM)  research  grew  out  of  a  perceived  need  for  understanding  how  people  control 
dynamic,  complex,  real-world  systems.  DDM  has  describable  characteristics  and,  with  some  unavoidable  sacrifice 
of  realism,  is  suitable  for  study  in  a  laboratory  setting  through  the  use  of  complex  computer  simulations  commonly 
called  ‘micro worlds’.  This  paper  presents  a  taxonomic  definition  of  DDM,  an  updated  review  of  existing 
microworlds  and  their  characteristics,  and  a  set  of  cognitive  demands  imposed  by  DDM  tasks.  Although  the  study  of 
DDM  has  garnered  little  attention  to  date,  we  believe  that  both  technological  advancement  and  the  relationships 
between  DDM  and  naturalistic  decision  making,  complex  problem  solving,  and  general  systems  theory  have  made 
DDM  a  viable  process  by  which  to  study  how  people  make  decisions  in  dynamic,  real-world  environments. 
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Gonzalez,  C.,  &  Wimisberg,  J.  (2007).  Situation  Awareness  in  Dynamic  Decision-Making:  Effects  of  Practice  and 
Cognitive  Abilities.  Journal  of  Cognitive  Engineering  and  Decision  Making,  1(1),  56-74. 
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Summary  abstract: 

The  goal  of  this  research  is  (a)  to  investigate  the  effects  of  task  practice  on  situation  awareness  (SA);  (b)  to 
investigate  how  cognitive  ability  -  in  particular,  working  memory  -  moderates  this  practice  effect  on  SA;  and  (c)  to 
investigate  effects  of  the  SA  measurement  procedure  (covering  or  uncovering  the  display  while  queries  are 
answered).  Task  practice  and  working  memory  influence  SA.  However,  the  dynamics  of  the  relationship  between 
working  memory  and  SA  over  time  are  not  well  understood,  particularly  with  regard  to  using  different  SA  measures. 
This  research  reports  an  experiment  in  which  different  SA  measurement  methods  were  used  while  participants 
played  a  computer  simulation  over  several  days.  SA  was  measured  using  two  query  methods:  a  covered  method  in 
which  questions  were  asked  while  the  display  was  blanked  out,  and  an  uncovered  method  in  which  questions  were 
asked  while  the  display  was  shown.  A  working  memory  measure  was  collected  from  participants.  Results  indicate 
that  the  relationship  between  working  memory  and  SA  diminishes  with  task  practice  and  differs  between  a  covered 
and  uncovered  method.  We  also  found  that  SA  improvement  with  practice  depends  on  the  way  in  which  SA  is 
measured.  We  discuss  the  implications  of  these  findings  for  understanding  SA  development,  SA  measurement,  and 
the  relationship  of  working  memory  in  this  process.  The  results  from  this  study  have  clear  implications  for  systems 
design,  for  the  design  of  learning  aids,  and  for  SA  measurement. 


Gonzalez,  C.,  &  Wimisberg,  J.  K.  (2003).  Can  I  Learn  to  Be  Aware?  Learning  Situation  Awareness  in  Dynamic 
Environments.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology 
Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi, 
Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

This  paper  presents  an  experiment  to  assess  ability  to  learn  Situation  Awareness  (SA)  in  a  Dynamic  Decision 
Making  task.  Participants  used  a  simulation  of  a  task  called:  Water  Production  Plant  (WPP)  and  the  SAGAT 
methodology  to  assess  SA  over  18  trials  of  running  the  WPP  simulation.  Results  show  that,  although  participants 
learn  to  perform  better  in  the  task  they  do  not  improve  their  SA  over  time.  Also,  overall  SA  scores  and  performance 
were  not  correlated.  Learning  showed  significant  only  in  some  of  the  SAGAT  questions  but  not  over  aggregated 
data  per  SA  level.  The  SA  overall  score  in  level  3  is  significantly  different  from  the  other  levels.  When  controlling 
for  individual  differences  in  fluid  intelligence,  the  results  show  SA  learning.  After  analyzing  this  learning  effect  over 
the  three  SA  levels,  the  results  showed  that  only  SA  Level  3  accounts  for  the  learning. 


Gorman,  J.  C.,  Foltz,  P.  W.,  Kiekel,  P.  A.,  Martin,  M.  J.,  &  Cooke,  N.  J.  (2003).  Evaluation  of  Latent  Semantic 
Analysis-Based  Measures  of  Communications  Content.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 
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Summary  abstract: 
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Team  process  is  thought  to  mediate  team  member  inputs  and  team  performance.  Among  the  team  behaviors 
identified  as  process  variables,  team  communications  have  been  widely  studied.  We  view  team  communications  as  a 
team  behavior  and  also  as  team  information  processing,  or  team  cognition.  Within  the  context  of  a  Predator 
Uninhabited  Air  Vehicle  (UAV)  synthetic  task,  we  have  developed  several  methods  of  communications  content 
assessment  based  on  Latent  Semantic  Analysis  (LSA).  These  methods  include:  Communications  Density  (CD) 
which  is  the  average  task  relevance  of  a  team’s  communications,  Lag  Coherence  (LC)  which  measures  taskrelevant 
topic  shifting  over  UAV  missions,  and  Automatic  Tagging  (AT)  which  categorizes  team  communications.  Each 
method  is  described  in  detail.  CD  and  LC  are  related  to  UAV  team  performance.  AT-human  is  comparable  to 
human-human  agreement  on  content  coding.  The  results  are  promising  for  the  assessment  of  teams  based  on  LSA 
applied  to  communication  content. 


Graham,  J.,  Gonzalez,  C.,  &  Doyle,  M.  (2003).  Using  Communication  Patterns  in  the  Design  of  an  Adaptive 
Organizational  Structure  for  Command  and  Control.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

What  type  of  analysis  should  be  used  to  inform  design  when  both  the  future  organizational  structure  and  the 
experimental  simulation  are  tenuous?  Neither  the  application  of  a  qualitative  process  tracing  method  nor  low-level 
quantitative  organizational  designs  are  warranted.  We  hypothesized  the  analyses  of  high-level  communication 
patterns  in  a  role-playing  exercise  of  a  future  organization  structure  would  yield  results  that  could  both  inform 
organizational  design  and  shape  iterative  experimental  designs.  This  study  summarizes  the  comparison  between 
communication  patterns  in  an  envisioned  organizational  structure  and  the  actual  patterns  of  information  exchange  of 
experienced  military  participants  role-playing  staff  members  in  a  future  organizational  design.  The  comparison 
between  the  hypothesized  and  actual  communication  performance  indicated  a  different  distribution  of 
communication  interaction  from  the  expected.  These  results  help  guide  both  the  future  organizational  concept  as 
well  as  next  iteration  experiments. 


Graham,  J.,  Schneider,  M.,  Bauer,  A.,  Bessiere,  K.,  &  Gonzalez,  C.  (2004).  Shared  Mental  Models  in  Military 
Command  and  Control  Organizations:  Effect  of  Social  Network  Distance.  In  In  Proceedings  of  48th  Annual  Meeting 
of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  presents  an  investigation  on  the  relationship  between  social  network  distances  and  shared  mental  models 
in  military  command  and  control  organizations.  Previous  research  has  shown  that  physical  distance  is  the  gold 
standard  for  high  performance  (Olson  &  Olson,  2000).  However,  social  network  distance  may  be  equally  or  more 
important,  as  social  network  graphs  inherently  take  into  account  the  group’s  context  and  environment  (Krackhart, 
1994).  We  conducted  this  research  on  a  new  56-member  command  and  control  organization  using  computer-based 
collaborative  tools  as  they  engaged  in  a  five  day  simulation  exercise.  As  military  command  and  control 
organizations  are  difficult  to  evaluate  based  on  outcome  and  performance,  we  chose  shared  mental  models  as  a 
proxy.  We  hypothesized  that  in  a  command  and  control  organization,  social  network  distance  and  physical  distance 
are  independent  of  one  another.  Further  we  hypothesized  that  social  network  distance  would  be  a  predictor  of  mental 
model  congruence.  We  found  that  there  is  a  very  weak  positive  correlation  between  social  network  distance  and 
physical  distance.  Further,  we  found  that,  controlling  for  physical  distance,  social  network  distance  is  a  predictor  of 
mental  model  congruence.  This  research  validates  that  high  frequency  of  communication,  mediated  by  computer 
based  collaborative  tools,  can  effectively  generate  shared  mental  models. 
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Graham,  J.,  Zheng,  L.,  &  Gonzalez,  C.  (2006).  A  Cognitive  Approach  to  Game  Usability  and  Design:  Mental  Model 
Development  in  Novice  Real-Time  Strategy  Gamers.  Cyber  Psychology  &  Behavior,  9(3),  361-366. 
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Summary  abstract: 

We  developed  a  technique  to  observe  and  characterize  a  novice  real-time-strategy  (RTS)  player’s  mental  model  as  it 
shifts  with  experience.  We  then  tested  this  technique  using  an  offthe-  shelf  RTS  game,  EA  Games  Generals.  Norman 
defined  mental  models  as,  “an  internal  representation  of  a  target  system  that  provides  predictive  and  explanatory 
power  to  the  operator.”  In  the  case  of  RTS  games,  the  operator  is  the  player  and  the  target  system  is  expressed  by  the 
relationships  within  the  game.  We  studied  five  novice  participants  in  laboratorycontrolled  conditions  playing  a  RTS 
game.  They  played  Command  and  Conquer  Generals  for  2  h  per  day  over  the  course  of  5  days.  A  mental  model 
analysis  was  generated  using  player  dissimilarity-ratings  of  the  game’s  artificial  intelligence  (AI)  agents  analyzed 
using  multidimensional  scaling  (MDS)  statistical  methods.  We  hypothesized  that  novices  would  begin  with  an 
impoverished  model  based  on  the  visible  physical  characteristics  of  the  game  system.  As  they  gained  experience  and 
insight,  their  mental  models  would  shift  and  accommodate  the  functional  characteristics  of  the  AI  agents.  We  found 
that  all  five  of  the  novice  participants  began  with  the  predicted  physical-based  mental  model.  However,  while  their 
models  did  qualitatively  shift  with  experience,  they  did  not  necessarily  change  to  the  predicted  functional-  based 
model.  This  research  presents  an  opportunity  for  the  design  of  games  that  are  guided  by  shifts  in  a  player’s  mental 
model  as  opposed  to  the  typical  progression  through  successive  performance  levels. 


Green,  C.  C.,  Griffin,  D.  E.,  Blascovich,  J.  J.,  Bradshaw,  J.  M.,  Bunce,  S.  C.,  Gannon,  J.,  Gazzaniga,  M.,  Loftus,  E., 
Moore,  G.  J.,  Moreno,  J.,  Rasure,  J.  R.,  Rintoul,  M.,  Schwade,  N.  D.,  Smith,  R.  L.,  Walch,  K.  S.,  &  Young,  A.  M. 
(2008).  Emerging  Cognitive  Neuroscience  and  Related  Technologies.  Report  of  the  Committee  on  Military  and 
Intelligence  Methodology  for  Emergent  Neurophysiological  and  Cognitive/Neural  Science  Research  in  the  Next 
Two  Decades  Standing  Committee  for  Technology  Insight — Gauge,  Evaluate,  and  Review  Division  on  Engineering 
and  Physical  Sciences  Board  on  Behavioral,  Cognitive,  and  Sensory  Sciences  Division  on  Behavioral  and  Social 
Sciences  and  Education  .  National  Research  Council.  Washington,  DC:  The  National  Academies  Press. 
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Summary  abstract: 

The  intelligence  community  (IC)  faces  voluminous  amounts  of  scientific  information  produced  and  available  on  a 
global  scale.  To  improve  analysis  of  the  information,  the  Technology  Warning  Division  (TWD)  of  the  Defense 
Intelligence  Agency’s  (DIA’s)  Defense  Warning  Office  (DWO)  asked  the  National  Research  Council  (NRC),  in 
2004,  to  establish  the  Committee  on  Defense  Intelligence  Agency  Technology  Forecasts  and  Reviews.  That 
committee  authored  the  report  Avoiding  Surprise  in  an  Era  of  Global  Technology  Advances.  Avoiding  Surprise 
provided  the  IC  with  a  technology  warning  methodology  not  previously  available  to  it  and  led  the  DIA  to  request 
that  the  NRC  establish  a  standing  committee  to  continue  to  provide  related  assistance.  In  May  2005,  the  Standing 
Committee  for  Technology  Insight — Gauge,  Evaluate,  and  Review  (TIGER)  was  established  to  assist  the  DWO  of 
DIA  in  formulating  future  studies  to  be  completed  by  NRC  ad  hoc  committees.  This  report  of  the  ad  hoc  Committee 
on  Military  and  Intelligence  Methodology  for  Emergent  Neurophysiological  and  Cognitive/Neural  Science  Research 
in  the  Next  Two  Decades  is  the  third  report  to  be  produced  under  the  purview  of  the  TIGER  Standing  Committee. 
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Grossman,  J.,  Voshell,  M.,  Tinapple,  D.,  Tittle,  J.,  &  Woods,  D.  D.  (2003).  Perceiving  Affordances  through  a 
Robot’s  Sensors.  In  In  Proceedings  of  6th  International  Conference  on  Naturalistic  Decision  Making.  Pensacola 
Beach,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  role  of  perception  in  cognition  has  been  a  basic  theme  in  NDM  work;  this  work  looks  at  how  to  enhance  the 
perception  of  the  affordances  of  scenes  for  remote  observers  working  through  a  robotic  platform.  One  of  the 
obstacles  is  seeing  affordances  through  a  robot's  sensors  As  robotic  system  developers  strive  to  achieve  a  certain 
level  of  autonomy,  they  underestimate  the  need  for  coordination  with  human  stakeholders.  This  has  been  generalized 
as: 

(Robin)  Murphy's  Law:  any  deployment  of  robotic  systems  will  fall  short  of  the  target  level  of  autonomy,  creating 
or  exacerbating  a  shortfall  in  mechanisms  for  coordination  with  human  problem  holders.  (Murphy  et  al.,  2002) 
Roboticists  don't  start  with  the  demands  of  the  settings  they  claim  to  assist.  When  you  start  with  the  field  you  see 
different  things  about  how  robots  and  people  need  to  work  together.  By  beginning  with  the  demands  of  complex 
operational  contexts  such  as  search  and  rescue  and  first  response  to  chemical/biological  incidents,  we  find 
alternative  models  for  the  relationship  of  people  and  robots.  Many  technological  advances  can  be  viewed  as  means 
for  perception  at  a  distance  or  action  at  distance.  New  capabilities  for  robotic  systems  are  a  major  step  forward 
within  this  tradition  of  coupling  people  to  scenes  at  a  distance.  In  this  view  robotic  capabilities  represent  new 
powers  for  human  problem  and  stake  holders  to  project  intent  at  a  distance. 

To  achieve  intent  at  a  distance  in  remote  explorations  of  an  environment,  a  robotic  platform  provides  an 
action/perception  stand  in  at  a  distance  for  its  handler.  This  de-couples  the  natural  dynamic  relationship  between 
properties  of  the  scene  being  explored  and  the  human  perceptual  system  of  the  remote  handler  (e.g.,  Cutting,  1986; 
Warren,  in  press).  The  decoupling  undermines  the  remote  observers  perception  of  affordances  in  the  scene  (Gibson, 
1979)  which  is  illustrated  by  recent  cases  of  HRC  where  remote  observers  experience  various  difficulties  in 
understanding  the  environment  being  traversed  by  a  robotic  platform  (Casper,  2002;  Darken,  Kempster,  and 
Peterson,  2001).  The  perceptual  ambiguities  (scale,  rate  of  motion,  keyhole  effect,  gaze  coordination)  make  clear 
that  seeing  though  a  remote  camera  is  not  the  equivalent  performance  as  having  a  human  observer  at  a  scene,  as 
perception  is  an  active  process  in  which  the  observer  causes  the  visual  image  to  change  by  performing  actions  in  the 
environment 

While  a  great  deal  of  work  has  addressed  creating  an  illusion  of  presence  for  remote  observers,  this  work  has  not 
addressed  the  fundamental  ambiguities  that  arise  in  remote  perception  or  how  to  enable  the  perception  of 
affordances  when  access  to  a  scene  is  mediated.  Our  research  demonstrates  several  initial  techniques  to  enhance 
raw  sensor  feeds  to  recover  what  was  lost  by  decoupling  the  human  perceptual  processor  from  the  environment 
being  explored. 


Grossman,  J.  B.,  Woods,  D.  D.,  &  Patterson,  E.  S.  (2007).  Supporting  the  Cognitive  Work  of  Information  Analysis 
and  Synthesis:  A  Study  of  the  Military  Intelligence  Domain.  In  In  Proceedings  of  51st  Annual  Meeting  of  the 
Human  Factors  and  Ergonomics  Society.  Baltimore,  MD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Information  Analysis  and  Synthesis  (henceforth  “IAS”)  is  a  type  of  cognitive  work  that  plays  a  key  role  in  many 
high-performance,  complex,  and  mission-critical  domains.  These  can  range  from  tactical  military  intelligence  to 
scientific  or  technological  forecasting,  business  and  financial  intelligence  to  national  strategic  counterterrorism,  and 
include  areas  as  disparate  as  geopolitical  policy  analysis  to  computer  network  intrusion  detection.  The  specific 
subject  of  this  study  is  the  military  intelligence  domain  as  one  instantiation  of  IAS.  Several  innovative  ethnographic 
and  cognitive  task  analysis  methods  were  used  to  observe  team-based  distributed  work  done  by  actual  domain 
practitioners.  The  main  investigative  effort  took  the  form  of  a  scaled-world  study,  leveraging  a  real  world  tactical 
intelligence  training  exercise  as  a  natural  laboratory  for  investigating  the  contrasts  between  weaker  and  stronger 
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IAS.  Specifically,  we  examined  the  role  of  instructors  in  providing  broadening  checks  to  the  team  analytic  process, 
and  mapped  the  findings  to  an  existing  framework. 


Gunawan,  L.  T.,  Voshell,  M.,  Oomes,  A.  H.  J.,  &  Woods,  D.  (2007).  Envisioning  Collaboration  at  a  Distance  for  the 
Evacuation  of  Walking  Wounded.  In  In  Proceedings  of  4th  International  ISCRAM  Conference  (B.  Van  de  Walle,  P. 
Burghardt  and  C.  Nieuwenhuis,  eds.),  Delft,  the  Netherlands. 

Key  Words: 
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Summary  abstract: 

The  "walking  wounded"  is  a  category  of  disaster  victims  that  can  help  themselves  in  finding  their  way  to  safety.  The 
problem  we  address  here  is  how  first  responders,  walking  wounded,  and  other  rescue  personnel  can  coordinate  their 
joint  activities  more  efficiently  in  order  to  accomplish  the  evacuation  as  quickly  as  possible.  We  focus  our  design  on 
the  "coordination  loops"  in  the  disaster  response  organization,  both  vertically  across  levels  of  authority,  and 
horizontally  among  responders  in  the  same  echelon.  In  our  envisioned  scenario  of  a  chemical  accident  we  identify 
the  most  important  interactions  through  which  activities  are  coordinated  that  are  crucial  for  a  successful  evacuation. 
We  propose  three  different  “coordination  devices”  that  can  be  used  by  the  walking  wounded,  the  rescuers  in  the 
fields,  and  the  people  in  the  command  center.  We  believe  our  approach,  explicitly  designing  support  systems  for 
coordination  first,  will  lead  to  important  improvements  in  the  daily  practice  of  disaster  response. 


Hancock,  J.  P.,  McGinnis,  S.,  &  Hillis,  D.  (2003).  Fast  Constraint  Satisfaction  Engine  for  Co-Evolutionary 
Approach  to  Tactical  Plan  Analysis.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003. 
Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland 
University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 
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Summary  abstract: 

From  the  original  Federated  Laboratories  program,  through  the  present  Advanced  Decision  Architectures  CTA, 

ARL  has  been  researching  the  application  of  Genetic  Algorithms  (GAs)  to  planning  applications.  The  current 
research  involves  the  exploration  of  applying  Co-Evolutionto  analyze  multi-sided,  non-linear  military  operations. 
The  extension  of  planning  research  to  multi-sided  operations  has  extended  the  requirements  on  plan  assessment 
beyond  those  of  standard  wargames. 

This  paper  details  the  design  of  the  Constraint  Satisfaction  based  Wargame  Operation  Processor  (WOPR).  WOPR  is 
the  faster-than-real-time  operation  plan  assessor  that  is  designed  to  support  ARL  co-evolution  planning  research. 
WOPR  is  composed  of  two  fundamental  parts,  the  Constraint  Solver  and  the  Wargame  Processor.  The  Vargame 
Processor  takes  one  output  of  the  Genetic  Algorithm,  (one  plan  per  faction),  and  plays  out  the  scenario  as  a  discrete 
event  wargame,  using  the  Constraint  Satisfaction  system  to  flesh-out  the  detailed  event  parameters.  The  Wargame 
Processor  walks  through  the  steps  of  the  faction's  plans  in  event  times  pace,  selecting  the  appropriate  constraints  to 
apply.  Factional  Plan  elements,  andforce  statuses  are  passed  to  the  Constraint  Solver  to  process  in  context.  All  status 
details  are  maintained  to  facilitate  review  by  human  planners  and  scoring  by  the  machine.  The  scoring  system  is 
highly  reconfigurable  to  support  the  experimentation  needs  of  ARL. 

Also  discussed  is  the  design  of  the  Constraint  Satisfaction  Integrated  Development  Environment  (CSIDE).  CSIDE  is 
a  tool  that  was  rapidly  created  to  support  the  research  development  of  an  integrated  set  of  Constraint  Sets  (CS).  The 
system  supports  visual  representation  of  Constraint  Set  elements:  Variables,  Generators,  and  Constraints,  as  well  as 
their  various  interconnections.  When  the  user  has  authored  a  Constraint  Set,  CSIDE  allows  the  user  to  setup 
XMLfiles  of  test  inputs,  and  to  run  them  against  the  CS  under  development.  CSIDE  is  useful  in  the  original 
development  of  Constraint  Sets  and  their  extensional  development  and  testing. 
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Hancock,  P.  A.,  &  Diaz,  D.  D.  (2002).  Ergonomics  as  a  Foundation  for  a  Science  of  Purpose.  Theoretical  Issues  in 
Ergonomic  Science,  3(2),  115-123. 
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Summary  abstract: 

The  argument  here  is  at  once  simple  and  profound.  The  first  premise  is  that  technology  is  the  most  powerful 
'shaping'  force  on  the  planet  today.  The  second  premise  is  that  individual  impact  on  this  force  is  most  evident  at  the 
humanmachine  interface.  A  corollary  of  the  second  premise  is  that  unity  in  intention  is  to  be  found  in  the  aggregate 
of  individual  actions.  Ergonomics,  as  the  pursuit  that  mediates  between  operator  and  technology,  is,  therefore,  the 
first  step  along  the  path  to  a  science  of  intention  as  expressed  in  'world'  changes.  Some  initial  observations  on  the 
integration  of  existing  behavioural  theories  represent  the  first  step  along  this  royal  road. 


Hancock,  P.  A.,  &  Szalma,  J.  L.  (2003).  The  Future  of  Neuroergonomics.  Theoretical  Issues  in  Ergonomic  Science, 
4(1-2),  238-249. 
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Brain  and  behaviour;  ergonomics;  ethics;  human  factors;  intentionality;  neuroscience. 

Summary  abstract: 

This  paper  addresses  the  theoretical,  the  philosophical,  and  the  ethical  considerations  associated  with  the  advent  and 
future  of  neuroergonomics.  These  issues  will  demand  increasing  attention  as  knowledge  of  the  linkage  of  brain 
function  to  technology-based  action  improves  from  its  current  coarse-grained  level  to  a  more  fine-grained 
understanding.  These  developments  promise  to  open  extraordinary  opportunities  for  improved  human-machine  and 
human-human  interaction,  and  represent  the  next  major  step  in  human-machine  evolution.  The  social  and 
psychological  implications  of  these  changes,  however,  must  be  considered  if  abuse  of  this  conception  is  to  be 
circumvented. 


Hancock,  P.  A.,  Weaver,  J.  L.,  &  Parasuraman,  R.  (2002).  Sans  Subjectivity,  Ergonomics  Is  Engineering. 
Ergonomics,  45(14),  991  -  994. 
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Summary  abstract: 

We  much  admire  and  agree  with  many  of  the  points  and  issues  that  Annett  has  raised  and  believe  his  work  has  great 
value  for  both  researchers  and  practitioners  in  ergonomics  and  allied  ®elds.  We  also  believe  that  Annett’ s  paper  will 
join  others  in  giving  the  he  to  the  frequent  assertion  that  ergonomics  is  an  atheoretical  ®eld,  an  ’appliance  science’ 
(Hancock  1997),  not  ®t  for  the  high  company  of  theoretical  psychology  and  at  best  a  poor  cousin  to  higher  forms  of 
engineering.  Annett’s  rich  theoretical  observations,  in  contrast,  begin  to  bring  to  the  fore  crucial  philosophical 
questions  that  underpin  the  ergonomics  enterprise. 
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Somewhat  ironically,  Annett  highlights  the  central  role  of  theory  in  ergonomics  by  discussing  the  use  of  subjective 
measures,  which  have  often  been  criticized  as  being  fundamentally  fawed  and  unscientific,  indeed  even  by  some  of 
the  founders  of  the  discipline,  as  Annett  points  out.  We  strongly  agree  with  Annett’s  main  points  that  all  empirical 
observations  have  a  subjective  component,  and  that  subjective  methods,  when  developed  and  applied  appropriately 
in  accordance  with  accepted  scienti®c  criteria,  have  an  important  place  in  ergonomics.  We  raise  two  additional 
issues,  however,  where  we  believe  Annett  has  not  gone  far  enough  and  as  commentators  we  solicit  his  further 
insight  into  these  matters. 


Harrison,  C.  L.,  &  Gillan,  D.  J.  (2005).  The  Role  of  Motion  in  Object  Recognition.  In  In  Proceedings  of  49th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Do  motion  cues  influence  object  recognition  when  contour  information  is  available?  Three  experiments  examined 
four  motion  conditions  for  a  variety  of  objects  (no  motion,  random  motion,  atypical  motion,  and  typical  motion) 
when  contour  information  was  also  available.  A  typical  motion  pattern  was  one  that  would  normally  be  associated 
with  the  moving  object,  whereas  atypical  motion  involved  a  regular  motion  pattern  that  was  typical  for  one  object  in 
the  set  of  15  used  in  the  experiments,  but  wasn't  associated  with  the  object  in  motion.  In  Experiments  1  and  2,  the 
objects  were  made  difficult  to  recognize,  by  eliminating  vertices  and  by  using  small  representations,  respectively.  In 
Experiment  3,  large,  complete  contour  wire  frame  pictures  were  used.  In  all  experiments,  recognition  speed  and 
accuracy  were  best  for  the  typical  motion  condition  and  second  best  for  the  atypical  motion.  With  easily-recognized 
objects,  random  motion  led  to  faster  recognition  than  no  motion,  whereas,  with  difficult  recognition,  random  motion 
led  to  slower  response  times  than  no  motion.  The  results  are  interpreted  with  a  three-process  model.  Applications  to 
the  design  of  computer  icons,  signage,  and  camouflage  are  discussed. 


Hillis,  D.,  Barnes,  M.,  Suantak,  L.,  Schlabach,  J.,  Chandrasekaran,  B.,  Josephson,  J.  R.,  &  Carroll,  M.  (2003). 
Collaborative  Visualization  Tools  for  Courses  of  Action  (Coa)  in  Antiterrorist  Operations:  Combining  Coevolution 
and  Pareto  Optimization.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and 
Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University 
College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

This  paper  describes  a  combination  of  coevolution,  specialized  simulation,  Pareto  optimization,  and  interactive 
visualization  to  aid  the  investigation  of  the  system  dynamics  in  Stability  and  Support  Operations  (SASO)  situations 
and  to  assist  in  the  choice  of  favorable  courses  of  action  (COA).  This  system,  which  is  partially  completed,  will 
consist  of  four  parts:  a  coevolution  engine,  a  wargamer  specialized  for  SASO  operations,  a  Pareto  filter  and 
graphical  COA  comparison  tool,  and  a  COA  scenario  visualization  tool.  Once  the  coevolution  engine  produces  and 
selects  CO  As  based  on  the  scores  provided  by  the  wargamer,  the  Pareto  filter  excludes  distinctly  suboptimal  friendly 
CO  As.  The  narrowed  set  of  favorable  COAs  is  returned  to  the  coevolution  engine  to  generate  more  COAs,  or  is 
passed  to  the  COA  comparison  tool  so  that  human  analysts  can  investigate  the  determinants  of  favorable  outcomes, 
understand  the  tradeoffs,  and  further  narrow  the  selection,  possibly  to  a  single  COA,  chosen  for  further 
investigation.  The  COA  scenario  visualization  tool  enables  analysts  to  watch  and  meter  the  events  of  individual 
scenarios,  facilitating  analysis  and  understanding.  This  paper  discusses  the  functions  of  each  of  the  components,  and 
how  they  can  function  together  to  support  a  commander’s  analysis  and  decision-making  of  a  given  scenario. 


Ho,  C.-Y.,  Nikolic,  M.  I.,  Waters,  M.  J.,  &  Sarter,  N.  B.  (2004).  Not  Now:  Supporting  Interruption  Management  by 
Indicating  the  Modality  and  Urgency  of  Pending  Tasks.  Human  Factors,  43(3),  399-409. 
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Summary  abstract: 

Operators  in  complex  event-driven  domains  must  coordinate  competing  attentional  demands  in  the  form  of  multiple 
tasks  and  interactions.  This  study  examined  the  extent  to  which  this  requirement  can  be  supported  more  effectively 
through  informative  interruption  cueing  (in  this  case,  partial  information  about  the  nature  of  pending  tasks).  The  48 
participants  performed  a  visually  demanding  air  traffic  control  (ATC)  task.  They  were  randomly  assigned  to  1  of  3 
experimental  groups  that  differed  in  the  availability  of  information  (not  available,  available  upon  request,  available 
automatically)  about  the  urgency  and  modality  of  pending  interruption  tasks.  Within-subject  variables  included 
ATC-related  workload  and  the  modality,  frequency,  and  priority  of  interruption  tasks.  The  results  show  that  advance 
knowledge  about  the  nature  of  pending  tasks  led  participants  to  delay  visual  interruption  tasks  the  longest,  which 
allowed  them  to  avoid  intramodal  interference  and  scanning  costs  associated  with  performing  these  tasks 
concurrently  with  ATC  tasks.  The  3  experimental  groups  did  not  differ  significantly  in  terms  of  their  interruption 
task  performance;  however,  the  group  that  automatically  received  task-related  information  showed  better  ATC 
performance,  thus  experiencing  a  net  performance  gain.  Actual  or  potential  applications  of  this  research  include  the 
design  of  interfaces  in  support  of  attention  and  interruption  management  in  a  wide  range  of  event-driven 
environments. 


Ho,  C.-Y.,  &  Sarter,  N.  B.  (2004).  Supporting  Synchronous  Distributed  Communication  and  Coordination  through 
Multimodal  Information  Exchange.  In  In  Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  New  Orleans,  LA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Increasingly,  operators  in  complex  event-driven  domains,  such  as  the  military,  need  to  coordinate  their  goals  and 
activities  with  numerous  co-located  and  distributed  human  and  machine  agents.  One  promising  way  to  support  this 
requirement  is  the  introduction  of  multimodal  interfaces  that  afford  functions  such  as  increased  bandwidth, 
complementarity,  redundancy,  and  substitution.  To  inform  the  design  of  a  robust  multimodal  system,  the  present 
study  explored  natural  tendencies  for,  and  the  role  of  context  in,  modality  usage  in  the  context  of  simulated 
battlefield  operations.  Three  groups  of  three  ROTC  cadets/officers  each  completed  a  set  of  30-minute  scenarios.  The 
within-subject  variables  included  participant  location,  the  availability  of  radio  communication,  the  amount  of 
coordination  required  by  the  scenario,  and  the  tempo  of  operations.  Our  findings  show  that  participants  were  highly 
selective  in  their  use  of  multiple  modalities.  Multimodal  interaction  was  observed  primarily  in  the  context  of  spatial 
tasks  and  for  the  purpose  of  supporting  grounding,  complementarity,  and  disambiguation.  Joint  modality  usage 
patterns  evolved  over  time  within  groups  and  varied  as  a  function  of  factors  such  as  scenario  and  interface 
management  demands.  The  findings  from  this  study  provide  important  guidance  for  the  design  of  multimodal 
combined  HCI  and  CSCW  interfaces  and  show  that  multimodal  information  exchange  is  a  highly  effective  means  of 
coordination. 


Ho,  H.,  &  Jones,  L.  A.  (2004).  Material  Identification  Using  Real  and  Simulated  Thermal  Cues.  In  In  Proceedings  of 
26th  Annual  International  Conference  of  the  IEEE  Engineering  in  Medicine  and  Biology  Meeting,  pp.  2462-2465. 
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Summary  abstract: 
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The  objective  of  this  study  was  to  develop  a  thermal  display  for  simulating  the  thermal  cues  associated  with  making 
contact  with  materials  with  different  thermal  properties.  The  thermal  display  was  designed  based  on  a  semi-infinite 
body  model.  The  performance  of  subjects  using  this  display  was  compared  to  that  with  real  materials  in  a  material 
identification  experiment.  The  results  indicated  that  there  was  no  significant  difference  in  material  identification 
when  subjects  were  presented  with  real  or  simulated  materials.  The  decreases  in  skin  temperature  were  also 
comparable  in  the  two  experiments  and  were  related  to  the  contact  coefficient  of  the  material  palpated,  which  is 
consistent  with  the  semi-infinite  body  model.  These  findings  suggest  that  thermal  feedback  in  a  haptic  display  can 
facilitate  object  recognition  when  visual  cues  are  limited. 


Ho,  H.-N.,  &  Jones,  L.  A.  (2007).  Development  and  Evaluation  of  a  Thermal  Display  for  Material  Identification  and 
Discrimination.  ACM  Transactions  on  Applied  Perception,  4(2-  Article  13). 
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Categories  and  Subject  Descriptors:  G.3  [Mathematics  of  Computing]:  Probability  and  Statistics — Experimental 
design;  H.1.2 

[Model  and  Principles]:  User/Machine  System;  H.5.2  [Information  Interfaces  and  Presentation]:  User  Interfaces — 
Evaluation/ 

methodology,  haptic  I/O,  theory  and  methods,  user-centered  design;  J.2  [Computer  Applications]:  Engineering 
General  Terms:  Design,  Experimentation,  Human  factors,  Measurement,  Performance,  Theory 
Additional  Key  Words  and  Phrases:  Haptic  interface,  material  identification,  semi-infinite  body  model,  thermal 
display,  thermal 

feedback,  thermal  perception,  hand-object  interaction,  virtual  environment 
Summary  abstract: 

The  objective  of  this  study  was  to  develop  and  evaluate  a  thermal  display  that  assists  in  object  identification  in 
virtual  environments  by  simulating  the  thermal  cues  associated  with  making  contact  with  materials  with  different 
thermal  properties.  The  thermal  display  was  developed  based  on  a  semi-infinite  body  model.  Three  experiments 
were  conducted  to  evaluate  the  performance  of  the  display.  The  first  experiment  compared  the  ability  of  subjects’  to 
identify  various  materials,  which  were  presented  physically  or  simulated  with  the  thermal  display.  The  second 
experiment  examined  the  capacity  of  subjects  to  discriminate  between  a  real  and  simulated  material  based  on 
thermal  cues.  In  the  third  experiment,  the  changes  in  skin  temperature  that  occurred  when  making  contact  with  real 
and  simulated  materials  were  measured  to  evaluate  how  these  compare  to  theoretical  predictions.  The  results 
indicated  that  there  was  no  significant  difference  in  material  identification  and  discrimination  when  subjects  were 
presented  with  real  or  simulated  materials.  The  changes  in  skin  temperature  were  comparable  for  real  and  simulated 
materials  and  were  related  to  the  contact  coefficient  of  the  material  palpated,  consistent  with  the  semi-infinite  body 
model.  These  findings  suggest  that  a  thermal  display  is  capable  of  facilitating  object  recognition  when  visual  cues 
are  limited. 


Ho,  H.-N.,  &  Jones,  L.  A.  (2007).  Monitoring  Skin  Temperature  During  Hand-Object  Interactions.  In  In  Proceedings 
of  InfraMation  Conference  2007  (pp.  1-8). 

Key  Words: 
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Summary  abstract: 

An  IR  thermal  measurement  system  was  created  to  monitor  skin  temperature  during  hand-object  interactions.  The 
layout  and  optical  arrangement  of  the  system  allowed  for  the  measurement  of  skin  temperature  across  the  fingerpad 
during  contact,  together  with  contact  force  and  contact  area.  An  IR  model  was  developed  to  compensate  for  the 
effects  of  the  optical  components  that  were  placed  between  the  fingerpad  and  IR  camera.  Based  on  this  model,  the 
measured  thermal  energy  could  be  converted  into  a  temperature  distribution  on  the  fingerpad  during  contact.  The 
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performance  of  this  system  was  evaluated  with  calibration  tests  and  the  results  indicated  that  this  system  was 
capable  of  providing  accurate  temperature  measurements  without  the  limitations  imposed  by  conventional  thermal 
sensors.  Measurements  obtained  with  this  system  can  be  used  to  evaluate  the  validity  of  models  used  to  predict  the 
changes  in  skin  temperature  during  hand-object  interactions. 


Ho,  H.-N.,  &  Jones,  L.  A.  (2008).  Modeling  the  Thermal  Responses  of  the  Skin  Surface  During  Hand-Object 
Interactions.  Journal  of  Biomechanical  Engineering.  ASME,  130,  21005-1  -  21005-8. 
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Summary  abstract: 

The  objective  of  this  research  is  to  analyze  and  model  the  decreases  in  skin  temperature  when  the  hand  makes 
contact  with  an  object  at  room  temperature  so  that  thermal  feedback  can  be  incorporated  into  haptic  displays.  A 
thermal  model  is  proposed  that  predicts  the  thermal  responses  of  the  skin  and  object  suiface  as  well  as  the  heat  flux 
exchanged  during  hand-object  interactions.  The  model  was  evaluated  by  comparing  the  theoretical  predictions  of 
temperature  changes  to  those  experimentally  measured  using  an  infrared  thermal  measurement  system.  The  thermal 
measurement  system  was  designed  to  overcome  the  limitations  imposed  by  contact  thermal  sensors,  and  was  able  to 
measure  skin  temperature  during  contact,  together  with  the  contact  area  and  contact  force.  The  experimental  results 
indicated  that  over  the  pressure  range  of  0.73-10.98  kPa,  changes  in  skin  temperature  were  well  localized  to  the 
contact  area  and  were  affected  by  contact  pressure.  The  pressure  in  turn  influenced  both  thermal  contact  resistance 
and  blood  flow.  Over  the  range  of  contact  forces  typically  used  in  manual  exploration,  blood  perfusion  and 
metabolic  heat  generation  do  not  appear  to  have  a  significant  effect  on  the  skin  s  thermal  responses.  The  theoretical 
predictions  and  the  measured  data  were  consistent  in  characterizing  the  time  course  and  amplitude  of  the  skin 
temperature  change  during  contact  with  differences  typically  being  less  than  10C  between  the  two  for  pressures 
greater  than  4  kPa.  These  findings  indicate  that  the  proposed  thermal  model  is  able  to  characterize  and  predict  the 
skin  temperature  responses  during  hand-object  interactions  and  could  be  used  in  a  thermal  display  that  simulates  the 
properties  of  different  materials. 


Hoffman,  R.  R.,  &  Hayes,  P.  J.  (2004).  The  Pleasure  Principle.  IEEE:  Intelligent  Systems,  86-89. 
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Summary  abstract: 

The  list  of ‘‘‘concepts  that  psychology  really  can’t  do  without”  includes  such  notions  as  neuronal  connectionism, 
degrees  of  consciousness,  mental  representation  of  information,  and  dissociation.  Of  the  pantheon  of  contributors  to 
the  history  of  psychology,  Aristotle  outranks  all  others  in  terms  of  the  number  of  critical  concepts  he  introduced, 
including  the  notion  of  the  association  of  ideas,  the  law  of  frequency  and  the  affiliated  concept  of  memory  strength, 
the  notion  of  stage  theories  of  development,  the  idea  of  distinguishing  types  of  mental  processes  or  faculties,  the 
idea  of  scales  of  nature  and  comparisons  between  humans  and  animals,  and  last  but  not  least,  the  Pleasure  Principle. 


Hoffman,  R.  R.  (2007).  Expertise  out  of  Context:  Papers  from  the  Sixth  International  Conference  on  Naturalistic 
Decision  Making.  Mahwah,  NJ:  Lawrence  Erlbaum  Associates,  Inc.  (in  production). 
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Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book 

Summary  abstract: 

The  main  theme  of  this  volume  is  captured  in  its  title.  The  authors  were  encouraged  to  discuss  topics  in  Naturalistic 
Decision  making  (NDM),  expertise  studies,  and  cognitive  systems  engineering,  with  an  eye  toward  questions  of 
what  happens  when  domain  practitioners  are  forced,  for  one  reason  or  another,  to  work  outside  of  their  comfort 
zone. 


Hoffman,  R.  R.  (2008).  Human  Factors  Contributions  to  Knowledge  Elicitation.  Human  Factors  (50th  Anniversary 
Special  issue),  50,  481-488. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Objective:  The  objective  of  this  article  is  to  lay  out  contributions  of  human  factors  to  knowledge  elicitation  (KE) 
methodology. 

Background:  The  background  is  historical,  dating  to  about  1985,  and  involves  the  convergence  of  expert  systems 
with  applied  psychology  and  cognitive  psychology. 

Method:  The  method  is  a  literature  review,  focusing  on  past  issues  of  Human  Factors. 

Results:  Human  factors  researchers  have  contributed  significantly  to  KE  methodology.  However,  KE  methodology 
“belongs  to”  a  number  of  communities  of  practice  and  has  applications  that  transcend  individual  disciplines. 
Conclusion:  Knowledge  elicitation,  thought  of  as  a  kind  of  cognitive  task  analysis,  grows  in  importance  with  the 
increasing  use  of  information  technology  to  form  complex  sociotechnical  work  systems  and  the  increasing 
importance  of  expertise  to  knowledge-based  organizations. 

Application:  I  discuss  some  open  issues  for  further  research  and  methodological  investigation. 


Hoffman,  R.  R.  (2009).  Cognitive  Task  Analysis  for  Creating  Macrocognitive  Work  Systems.  In  Invited 
Presentation  at  the  Annual  Conference  of  the  International  Technology  Alliance,  sponsored  by  the  Army  Resarch 
Laboratory  and  The  Ministry  of  Defense  (Unted  Kingdom).  University  of  Maryland  Conference  Center,  College 
Park,  MD,  September  2009. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  talk  will  describe  Cognitive  Task  Analysis  (CTA)  methods  that  have  been  proven  useful  in  the  design  of 
sociotechnical  systems.  A  number  of  methods  will  be  described,  including  the  Critical  Decision  Method,  the 
Cognitive  Modeling  Procedure,  Workplace  Observation,  and  Concept  Mapping.  The  discussion  will  highlight 
strengths  and  weaknesses  of  methods,  and  appropriate  combinations  of  methods.  Applications  of  CTA  will  also  be 
described  in  this  presentation,  converging  on  specific  ITA  needs  and  issues  that  require  the  empirical  study  of 
cognition  in  context.  The  ITA  faces  an  "envisioned  world  problem"  requiring  choices  and  combinations  of  CTA 
methods  appropriate  for  the  investigation  of  the  challenge  questions. 


Hoffman,  R.  R.,  Amin,  R.,  Brents,  C.,  Bunch,  L.,  Carff,  R.,  Endsley,  M.,  Hancock,  P.  A.,  Klein,  G.,  Moon,  B.,  Marx, 
M.,  McDermott,  P.,  &  Woods,  D.  (2009).  Theories  and  Methods  for  Measurement  and  Metrics  for  Advanced 
Decision  Architectures  Prepared  as  a  deliverable  to  the  ARL  Advanced  Decision  Architectures  Collaborative 
Alliance  Cooperative  Agreement  DAAD19-01-2-0009.  Prepared  by  University  of  West  Florida/Institute  for  Human 
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and  Machine  Cognition, with  contributions  by  Alion  Science  MAAD  Operation  Klein  Associates  Division  of  ARA 
Ohio  State  University  Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Report 

Chapter  1:  About  this  ARLADA  Task 

Chapter  2:  Basics  of  Measurement  and  Metrics 

Chapter  3 :  Human  Performance  Measurement 

Chapter  4:  Hedonic  Measurement 

Chapter  5:  Range  Statistics  and  Learnability 

Chapter  6:  Modeling  Range  Statistics 

Chapter  7 :  Range  Statistics  and  Universal  Metrics 

Chapter  8:  Measurement  and  Cognitive  Task  Analysis 

Chapter  9:  Macrocognitive  Measurement 

Chapter  10:  Measurement  at  the  "System  Level" 

Summary  abstract: 

This  report  expresses  the  findings  from  an  ARLADA  Task  on  developing  measures  and  metrics  that  can  be  broadly 
applied  in  the  analysis  of  any  "advanced  decision  architecture."  That  is,  the  evaluation  of  intelligent  technologies 
and  decision  aides  and  the  cognitive  work  that  is  shaped  by  the  software  tools  the  architecture  provides.  This 
ARLADA  Task  effort  had  a  broad  goal  of  establishing  foundations  for  measurement,  especially  measurement  for 
cognitive  work  in  sociotechnical  work  systems.  Chapters  cover  the  basics  of  measurement  and  measurement  theory, 
human  performance  measurement,  experimental  methodology  in  the  formation  of  measures  and  metrics,  measures  to 
evaluate  cognitive  work  and  measures  to  evaluate  performance  at  the  "system  level."  While  presenting  foundational 
ideas,  the  report  also  presents  new  ideas  and  mathematical  treatments,  intended  to  significantly  extend  measurement 
and  metrics  beyond  traditions  into  new  realms.  In  particular,  this  Report  describes  a  new  field  of  "Range  Statistics," 
which  presents  general  measures  for  assessing  the  learnability  of  softwaresupported  work  of  any  kind,  and  a  means 
for  measuring  the  resilience  of  cognitive  work.  These  are  new  measures  that  can  be  taken  all  the  way  to  "Universal 
Metrics"  for  the  evaluation  of  the  quality  of  cognitive  work  of  any  kind.  The  spirit  of  this  ARLADA  Task  effort  is 
captured  by  the  quotation  attributed  to  former  Secretary  of  Defense  Robert  MacNamara:  "We  need  to  stop  making 
what  is  measurable  important  and  start  making  what  is  important  measurable."  Cognitive  systems  engineering  seeks 
understanding  of  very  high  orders  of  cognitive  activity  and  work  complexity,  matched  by  very  high  levels  of 
importance  to  society.  The  view  that  guided  this  ARLADA  Task  was  that  the  measurement  of  cognitive  work  must 
be  taken  to  entirely  new  levels,  ones  that  can  address,  for  example,  concerns  such  as  team  cognitive  work  (and  not 
just  one  person-one  machine  interaction),  and  address  the  important  trade-offs  in  cognitive  work  that  emerge  only 
when  we  consider  action  at  the  system  level — which  is,  after  all,  where  the  final  pay-off  for  any  investment  is 
realized. 


Hoffman,  R.  R.,  Bradshaw,  J.  M.,  Hayes,  P.  J.,  &  Ford,  K.  M.  (2003).  The  Borg  Hypothesis.  IEEE  Intelligent 
Systems,  (September/October),  pp.  73-75. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

What  if  intelligent  computing  were  centered  inside  humans?  This  essay’s  title  is  inspired  by  the  nemesis  of  Jean-Luc 
Picard,  captain  of  the  starship  Enterprise  in  the  television  series  Star  Trek:  The  Next  Generation.  The  Borg  are — or 
should  we  say  “is” — a  species  consisting  of  organic  beings  symbiotically  merged  with  technology.  Each  individual 
Borg  is  laden  with  all  manner  of  appliances,  ranging  from  laser  eyeballs  to  appendages  resembling  drill  presses  to 
computational  and  communication  devices  implanted  in  their  nervous  systems.  The  Borg  is  a  collective,  meaning 
that  they — or  it —  possess  a  single  mind.  That  Borg  mind  has  the  single  intent  of  “assimilating”  all  organic  species 
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into  the  collective.  Assimilation  involves  first  injecting  nanoprobes  that  thoroughly  transform  the  organic  being 
down  to  the  molecular  level,  then  grafting  on  the  various  appliances  (or  else  growing  them  de  novo  like  so  many 
cloned  carrots  in  a  hydroponic  garden).  Wending  their  way  through  the  galaxy  in  huge  Rubik  Cube-like  vehicles,  the 
Borg  assimilate  entire  planets  at  a  time  and  carve  up  starships  as  if  they  were  roast  beef,  making  them  (it)  an 
especially  nasty  adversary. 

In  our  real  world,  we  already  routinely  replace  hip  joints  with  titanium  and  inner-ear  structures  with  microcircuits; 
we  can  carry  telephones  comfortably  on  our  heads,  and  Web-enabled  eyeglasses  can  augment  our  view  of  reality.  To 
counter  the  effects  of  drowsiness  or  inattention,  DaimlerChrysler  is  developing  prototypes  that  continuously  monitor 
drivers’  physical  and  mental  states,  while  DARPA’s  Augmented  Cognition  Program  is  planning  an  even  more 
ambitious  reach  to  “plug  in”  the  warfighter  of  the  future  (www.darpa.mil/ipto/programs/augcog/index.htm). 
Portending  an  even  braver  and  newer  world,  it’s  now  possible  to  insert  wires  into  a  person’s  nerves  to  control 
appliances.  We  can  even  send  such  signals  over  the  Internet,  where  they  are  decoded  by  computer  and  then  fed  into 
another  person’s  nervous  system.Human  bodies  are  getting  more  and  more  plugged  in.  It’s  not  easy  to  set  aside 
questions  of  ethics  and  choice.  It  is  not  even  possible.  However,  in  this  essay  we  simply  overlook  them  in  order  to 
work  toward  our  hypothesis.  To  do  that,  we  must  take  you  on  a  trip  into  space.  Our  argument  is  that  if  humanity 
decides  to  continue  human  exploration  of  space,  we  will  sooner  or  later — probably  sooner — be  forced  to  center 
some  intelligent  computing  inside  humans. 


Hoffman,  R.  R.,  &  Deal,  S.  V.  (2008).  Influencing  Versus  Informing  Design,  Part  1:  A  Gap  Analysis.  IEEE: 
Intelligent  Systems.  Human  Centered  Computing,  72-75. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

In  this  and  the  next  essay  in  this  department,  we  reflect  on  the  mismatch  that  can  occur  between  the  promise  of 
intelligent  technology  and  the  results  of  technological  interventions.  Cognitive  task  analysis  (CTA)  has  been  offered 
as  a  way  of  revealing  actual  work,  especially  macrocognition,  but  the  system  development  process  can  fail  to  bridge 
the  gap  between  the  products  of  CTA  and  the  needs  of  software  engineers.  This  is  the  first  of  two  essays  about 
bridging  this  gap.  In  this  essay  we  discuss  a  success  story  of  cognitive  systems  engineering  to  illustrate  a  product 
from  CTA  that  speaks  in  a  language  that’s  similar  to  that  of  software  designers.  The  second  essay  is  about  bridging 
the  gap  going  from  the  other  direction,  providing  designers  with  a  CTA  method  that  they  themselves  can  use  to 
ramp  up  their  understanding  of  end  users’  true  work  needs. 


Hoffman,  R.  R.,  &  Elm,  W.  C.  (2006).  Human-Centered  Computing  (Hcc)  Implications  for  the  Procurement 
Process.  IEEE:  Intelligent  Systems,  pp.  .21(1),  74-81. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
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Journal  Article 

Summary  abstract: 

Most  system  designers  and  human  factors  engineers  have  participated  in  projects  that  culminated  in  systems  that 
were  highly  constrained  by  short-term  cost  considerations.  In  the  procurement  of  information  processing  and 
intelligent  technology  for  complex  sociotechnical  domains,  the  focus  on  short-term  cost  considerations  at  the 
expense  of  human-centering  considerations  always  comes  with  a  hefty  price  down  the  road.  This  price  weighs  much 
more  heavily  on  users’  shoulders  than  on  those  of  the  technologists  or  project  managers. 
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Hoffman,  R.  R.,  Feltovich,  P.  J.,  Ford,  K.  M.,  Woods,  D.  D.,  &  Feltovich,  A.  (2002).  A  Rose  by  Any  Other  Name  ... 
Would  Probably  Be  Given  an  Acronym.  IEEE  Intelligent  Systems,  July-August,  72-79. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

In  this  essay,  we  concern  ourselves  with  characterizations  of  the  “new”  approaches  to  the  design  of  complex 
sociotechnical  systems,  and  we  use  a  biological  classification  scheme  to  organize  the  discussion.  Until  fairly 
recently  the  design  of  complex  sociotechnical  systems  was  primarily  known  as  cognitive  engineering  or  cognitive 
systems  engineering  (CSE),  a  term  introduced  in  the  1980s  to  denote  an  emerging  branch  of  applied  cognitive 
psychology. 

Research  focused  on  such  topics  as  human-computer  interaction,  the  psychology  of  programming,  display  design, 
and  user  friendliness.  Although  some  have  sought  to  make  the  term  cognitive  engineering  seem  less  of  an  oxymoron 
by  doing  work  that  somehow  looks  like  actual  engineering,  a  number  of  new  terms  have  emerged,  all  of  which 
might  be  considered  members  of  the  “genus”  Human-Centered  Computing.  Researchers,  research  organizations, 
funding  sources,  national  study  groups  and  working  groups,  and  even  entire  national  funding  programs  espouse 
these  approaches.  A  number  of  varieties  have  entered  the  judging  competition. 

This  variety  has  come  about  for  many  reasons.  Some  individuals  have  proposed  terms  to  express  views  that  they 
believe  are  new.  Others  have  proposed  terms  as  a  consequence  of  the  social  and  competitive  nature  of  science  and 
science  funding,  leading  to  turf  wars  and  the  need  for  individuals  to  win  awards  and  claim  niches  that  set  themselves 
and  their  ideas  apart  from  the  crowd.  The  obvious,  and  obviously  incorrect,  question  is,  “Which  term  is  the  right 
one?”  As  we  hope  to  suggest  in  this  essay,  this  question  is  rather  like  the  quest  for  the  blue  rose.  Using  the  rose 
metaphor,  and  taking  some  liberties  with  Latin,  we  organize  the  essay  around  a  set  of  “genuses”  into  which  the 
individual  “varieties”  seem  to  fall. 


Hoffman,  R.  R.,  &  Hanes,  L.  F.  (2003).  The  Boiled  Frog  Problem.  IEEE  Intelligent  Systems,  July,  pp.  68-71. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

They  say  that  a  frog  placed  in  boiling  water  will  protest  vigorously,  but  one  placed  in  cool  water  that  is  slowly 
brought  to  a  boil  will  languish  happily  until  it  is  ready  to  be  a  puree.  We  use  this  as  a  metaphor  to  discuss  an 
outstanding  problem  in  knowledge  management — a  problem  that  human-centered  computing  can  perhaps  address. 

In  all  complex  sociotechnical  workplaces,  including  those  in  government  and  industry,  knowledge  and  skill  have 
become  widely  recognized  as  increasingly  important  assets.  They  are  important  because  expertise  is  a  “must”  for 
proficient  performance  in  these  domains.  Furthermore,  this  importance  is  increasing  as  we  recognize  that  many  of 
the  most  knowledgeable  personnel  are  nearing  retirement,  and  there  are  adverse  consequences  associated  with  losing 
their  expertise.  An  outright  panic  attack  comes  when  an  organization  realizes  that  it  does  not  have  a  plan  for 
capturing  the  valuable  knowledge  about  to  be  lost.  Like  the  frog  languishing  happily,  management  will  languish 
happily  until  it  realizes  that  it  is  getting  into  “very  hot  water”  because  of  a  loss  of  expertise.  Widespread  recognition 
of  the  boiled  frog  problem  is  apparent  in  the  current  popularity  of  the  phrases  knowledge  preservation  and 
knowledge  management,  as  Gary  Klein  discussed  in  his  seminal  paper,  “Using  Knowledge  Engineering  to  Preserve 
Corporate  Memory.”  Many  organizations  have  discovered — either  the  hard  way  or  too  late — that  expert  wisdom  is  a 
corporate  asset.  For  example,  NASA  encountered  problems  owing  to  its  loss  of  engineering  expertise  from  the 
Apollo  era.  Similarly,  a  retiring  supervisor  at  a  large  soup  manufacturing  firm  possessed  a  unique  skill  for 
controlling  the  large  soup-making  machines.  On  the  verge  of  his  retirement,  the  company  realized  no  one  could  do 
what  he  did,  so  they  brought  in  a  team  of  knowledge  engineers  to  elicit  and  preserve  his  skills. 


169 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 


Hoffman,  R.  R.,  Hayes,  P.,  Ford,  K.  M.,  &  Hancock,  P.  (2002).  The  Triples  Rule.  IEEE:  Intelligent  Systems,  pp.  62- 
65. 

Key  Words: 
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Journal  Article 

Summary  abstract: 

Afundamental  stance  taken  in  human-centered  computing  is  that  information-processing  devices  must  be  thought  of 
in  systems  terms.  At  first  blush,  this  seems  self-evident.  However,  the  notion  has  a  long  history,  and  not  just  in 
systems  engineering. 


Hoffman,  R.  R.,  Klein,  G.,  &  Laughery,  K.  R.  (2002).  The  State  of  Cognitive  Systems  Engineering.  IEEE: 

Intelligent  Systems,  73-75. 

Key  Words: 
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Journal  Article 

Summary  abstract: 

The  widespread  introduction  of  the  personal  computer,  beginning  about  1970,  helped  spawn  the  field  of  inquiry 
called  cognitive  engineering,  which  concerns  itself  with  such  things  as  interface  design  and  user  friendliness. 

Since  then,  this  field  has  taught  us  many  important  things,  including  two  major  lessons. 

First,  the  road  to  user-hostile  systems  is  paved  with  designers’  user-centered  intentions.  Even  smart,  clever,  well- 
intentioned  people  can  build  fragile,  hostile  devices  that  force  the  human  to  adapt  and  build  local  kludges  and 
workarounds.  Worse  still,  even  if  you  are  aware  of  this  trap,  you  will  still  fall  into  it. 

Second,  technology  developers  must  strive  to  build  truly  human-centered  systems.  Machines  should  adapt  to  people, 
not  the  other  way  around.  Machines  should  empower  people.  The  process  of  designing  machines  should  leverage 
what  we  know  about  human  cognitive,  perceptual,  and  collaborative  skills. 


Hoffman,  R.  R.,  Lee,  J.  D.,  Woods,  D.  D.,  Shadbolt,  N.,  Miller,  J.,  &  Bradshaw,  J.  M.  (2009).  The  Dynamics  of 
Trust  in  Cyberdomains.  Institute  for  Human  and  Machine  Cognition;  IEEE  Intelligent  Systems,  pp.  5-11. 
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Summary  abstract: 

All  economic,  social,  and  legal  interactions  are  based  on  assumptions  that  individuals  can  verify  identities;  that  they 
can  rely  on  rules,  institutions,  and  normative  practice;  and  that  they  can  be  assured  that  their  private  space  will 
remain  protected.  Monitoring,  managing,  verifying,  auditing,  and  enforcing  these  assumptions  are  difficult  online. 


Hoffman,  R.  R.,  &  Lintern,  G.  (2006).  Chapter  12.  Eliciting  and  Representing  the  Knowledge  of  Experts.  In  A. 
Ericsson,  N.  Charness,  R.  Hoffman,  &  P.  Feltovich  (Eds.),  Cambridge  Handbook  on  Expertise  and  Expert 
Performance.  New  York:  Cambridge  University  Press. 
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Key  Words: 
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Book  Chapter/Section 

knowledge  elicitation,  expert  systems,  intelligent  systems,  methodology,  Concept  Maps,  Abstraction- 
Decomposition,  critical  decision  method 

Summary  abstract: 

The  transgenerational  transmission  of  the  wisdom  of  elders  via  storytelling  is  as  old  as  humanity  itself  During  the 
Middle  Ages  and  Renaissance,  the  Craft  Guilds  had  wellspecified  procedures  for  the  transmission  of  knowledge,  and 
indeed  gave  us  the  developmental  scale  that  is  still  widely  used:  initiate,  novice,  apprentice,  journeyman,  expert,  and 
master  (Hoffman,  1998).  Based  on  interviews  and  observations  of  the  workplace,  Denis  Diderot  (along  with  140 
others,  including  Emile  Voltaire)  created  one  of  the  great  works  of  the  Enlightenment,  the  17  volume  Encyclopedie 
(Diderot,  1751-1772),  which  explained  many  "secrets"  -  the  knowledge  and  procedures  in  a  number  of  tradecrafts. 
The  emergent  science  of  psychology  of  the  1700S  and  1800s  also  involved  research  that,  in  hindsight,  might 
legitimately  be  regarded  as  knowledge  elicitation  (KE).  For  instance,  a  number  of  studies  of  reasoning  were 
conducted  in  the  laboratory  of  Wilhelm  Wundt,  and  some  of  these  involved  university  professors  as  the  research 
participants  (Militello  &  Hoffman,  forthcoming).  In  the  decade  prior  to  World  War  I,  the  stage  was  set  in  Europe  for 
applied  and  industrial  psychology;  much  of  that  work  involved  the  systematic  study  of  proficient  domain 
practitioners  (see  Hoffman  &  Deffenbacher,  1992). 

The  focus  of  this  chapter  is  on  a  more  recent  acceleration  of  research  that  involves  the  elicitation  and  representation 
of  expert  knowledge  (and  the  subsequent  use  of  the  representations,  in  design).  We  layout  recent  historical  origins 
and  rationale  for  the  work,  we  chart  the  developments  during  the  era  of  first-generation  expert  systems,  and  then  we 
proceed  to  encapsulate  our  modern  understanding  of  and  approaches  to  the  elicitation,  representation,  and  sharing  of 
expert  knowledge.  Our  emphasis  in  this  chapter  is  on  methods  and  methodological  issues. 


Hoffman,  R.  R.,  Lintern,  G.,  &  Eitelman,  S.  (2004).  The  Janus  Principle.  IEEE:  Intelligent  Systems,  78  -  80. 
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Summary  abstract: 

The  real  Janus  (Ianua)  may  have  been  an  ancient  king  from  Greece  who  encouraged  his  (adopted  or  conquered) 
people  of  Latium  (now,  a  part  of  Italy)  to  acquire  the  skills  of  agriculture,  industry,  art,  and  religion.  Over  time,  the 
Roman  myth  arose  of  a  god  who  brought  all  things  into  the  world,  from  the  seasons  to  the  ways  of  civilization.  This 
god  of  gates  and  doorways  opened  and  closed  all  things,  including  the  gates  of  heaven.  Janus  was  symbolized  as  a 
two-faced  god,  one  face  being  that  of  a  youth,  the  other  of  an  elder.  In  his  left  hand,  the  youth  held  a  key  to  open  the 
gates.  In  his  right,  the  elder  held  a  scepter  to  master  all  comings  and  goings.  We  find  here  an  allegory  to  the 
apprentice-expert  continuum  in  the  acquisition  of  knowledge,  and  so  we  name  a  principle  of  human-centered 
computing  after  Janus.  This  principle  deals  with  the  distinction  between  performance  and  training,  and  its 
implications  for  intelligent  technologies. 


Hoffman,  R.  R.,  Marx,  M.,  Amin,  R.,  &  McDermott,  P.  L.  (2009).  Chapter  6.  How  Good  Is  That  New  Software 
Tool?  The  Mathematical  Modeling  of  Performance  Metrics.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced 
Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  I.  Collecting,  Processing  and 
Distributing  Battlefield  Information:  48hrBooks  by  Alion  Science  &  Technology  Corporation  under  contract  to  the 
U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and  Engineering 
Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology 
Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision 
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Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr- 
webtest. alionscience.com/AdaCta/ Agenda,  aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

This  chapter  presents  a  method  for  evaluating  performance  using  cognitively-inspired  Advanced  Decision 
Architectures  and  decision  support  tools.  The  method  can  aid  evaluators  at  test  centers  and  cognitive  scientists  who 
adopt  a  systems-level  approach  to  evaluation.  The  warfighter  can  benefit  by  receiving  tools  that  better  support 
cognitive  functioning  and  situation  awareness.  In  particular,  we  are  interested  here  in  measuring  the  leamability  of 
new  software-supported  work  processes  and  the  resilience  of  the  work  to  disruption. 


Hoffman,  R.  R.,  Marx,  M.,  Amin,  R.,  &  McDermott,  P.  L.  (2009).  Measurement  for  Evaluating  the  Leamability  and 
Resilience  of  Methods  of  Cognitive  Work.  Theoretical  Issues  in  Ergonomic  Science  (in  press). 
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cognitive  work,  performance  measurement,  leamability,  range  statistics,  technology  evaluation,  resilience 
Summary  abstract: 

Some  experiments  on  human-computer  interaction  are  aimed  at  evaluating  hypotheses  concerning  cognitive  work. 
Other  experiments  are  intended  to  evaluate  the  software  tools  that  shape  the  cognitive  work.  In  both  cases,  effective 
experimentation  is  premised  on  the  control  and  factorial  analysis  of  sources  of  variability.  This  entails  programs  of 
experimentation.  However,  sociotechnical  systems  are  generally  a  “moving  target”  in  terms  of  the  pace  of  change. 
Our  objective  was  to  create  a  general  approach  to  experimental  design  and  the  measurement  of  cognitive  work  that 
can  satisfy  the  requirements  for  experimentation  and  yet  can  also  provide  a  “fast  track”  to  the  evaluation  of 
software-supported  cognitive  work.  We  present  a  measure  called  i-bar,  which  is  the  inverse  of  the  mid-range.  The 
statistic  is  derived  from  data  on  trials-tocriterion  in  tasks  that  require  practice  and  learning.  We  interpret  this  single 
measure  as  a  conjoint  measurement  scale,  permitting  (a)  evaluation  of  sensitivity  of  the  principal  performance 
measure  (which  is  used  to  set  the  metric  for  trials  to  criterion),  (b)  evaluation  of  the  leamability  of  the  work  method 
(i.e.,  the  goodness  of  the  software  tool),  and  (c)  evaluation  of  the  resilience  of  the  work  method.  We  show  that  it  is 
possible  to  mathematically  model  such  order  statistics,  and  derive  methods  for  estimating  likelihoods.  This  involves 
novel  ways  of  thinking  about  statistical  analysis  for  discrete  non-Gaussian  distributions.  The  idea  and  method  we 
present  should  be  applicable  to  the  study  of  the  effects  of  any  training  or  intervention,  including  software 
interventions  designed  to  improve  legacy  work  methods  and  interventions  that  involve  creating  entirely  new 
cognitive  work  systems. 


Hoffman,  R.  R.,  Marx,  M.,  &  Hancock,  P.  (2008).  Metrics,  Metrics,  Metrics:  Negative  Hedonicity.  IEEE:  Intelligent 
Systems,  69-73. 
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Summary  abstract: 

Intelligent  technologies  such  as  performance  support  systems  and  decision  aids  represent  a  key  aspect  of  modern 
sociotechnical  systems.  When  new  tools  are  introduced  into  the  workplace,  they  represent  hypotheses  about  how 
cognitive  work  is  expected  to  change.  1,2  The  tacit  hypothesis  is  that  any  such  change  will  be  for  the  better. 


172 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 


performance  will  be  more  efficient,  and  decisions  will  be  improved —  that  is,  they’ll  be  made  faster  and  on  the  basis 
of  greater  evidence.  Experience  suggests  that  technological  interventions  sometimes  have  the  intended  positive 
effect.  However,  they  often  result  in  negative  effects,  including  unintended  cascading  failures  and  worker  frustration 
due  to  “user-hostile”  aspects  of  interfaces. 3 

Concern  is  rising  about  the  high  rate  of  software  procurement  failures  that  are  due  to  the  inadequate  consideration  of 
human  factors.  Recent  statistics  suggest  a  dismal  record,  representing  the  expenditure  of  billions  of  dollars  for 
technologies  that  are  unusable,  ineffective,  and  at  times  even  defunct. 4, 5  At  the  same  time,  funding  for  developing 
communication  and  information  technologies  has  reached  record  levels  (about  US$500  billion  in  the  mid-1990s). 6 
Recently  we’ve  seen  entire  government-sponsored  research  programs  with  titles  that  state  human-system  integration 
as  a  key  goal  for  new  technologies. 7  The  notorious  frustrations  and  failures  triggered  by  software  interventions  in 
the  workplace  have  led  to  a  significant  concern  in  the  software  engineering  community  with  evaluation, 8,9  including 
help  for  organizations  to  establish  metrics  for  “key  performance  indicators.”! 0,11 


Hoffman,  R.  R.,  &  McNeese,  M.  D.  (2009).  A  History  for  Macrocognition.  Journal  of  Cognitive  Engineering  and 
Decision  Making,  3,  97-110. 
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Summary  abstract: 

This  article  presents  a  historical  context  for  the  current  notion  of  macrocognition.  The  idea  of  macrocognition  has 
precedents  dating  to  the  first  decades  of  experimental  psychology.  The  distinction  between  macrocognition  and 
microcognition  instantiates  philosophical  issues  that  contrast  structuralism  with  functionalism  (naturalism  and 
holism).  This  is  one  of  the  thematic  “pendulum  swings”  across  the  history  of  psychology.  With  regard  to  both 
experimentation  and  computational  modeling,  the  issues  that  persist  across  history  can  be  interpreted  as  cautionary 
tales,  as  constraints  on  methodologies,  or  as  challenges. 


Hoffman,  R.  R.,  &  Militello,  L.  G.  (2009).  Perspectives  on  Cognitive  Task  Analysis:  Historical  Origins  and 
Modern  Communities  of  Practice  Boca  Raton,  FL:  CRC  Press/Taylor  and  Francis. 
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Summary  abstract: 

The  purpose  of  this  book  is  to  provide  a  context  for  understanding  Cognitive  Task  Analysis  (CTA)  in  terms  of  its 
historical  roots  and  recent  origins  in  the  traditional  disciplines  and  emerging  communities  of  practice.  In  the  chapters 
of  this  book,  we  show  how  CTA  came  to  be  what  it  is  today.  We  show  where  the  many  CTA  methods  have  come 
from  and  what  they  are  intended  to  do.  We  also  provide  integration  by  showing  the  relations,  dovetailings,  common 
themes,  and  common  purposes  of  the  various  disciplines  that  have  contributed  to  our  modern  understanding  of  CTA. 
We  especially  focus  on  the  convergence  with  regard  to  the  design  of  information  systems,  including  intelligent 
technologies. 

This  book  is  intended  to  be  a  companion  to  Working  Minds:  A  Practitioner's  Guide  to  Cognitive  Task  Analysis 
(Crandall,  Klein,  &  Hoffman,  2006),  which  goes  into  detail  about  the  actual  conduct  of  CT  A  methods.  This  book 
can  also  be  seen  as  having  a  companion-Neville  Moray's  History  and  Scope  of  Human  Factors  (2005),  which 
reprints  many  of  the  classic  writings  that  are  cited  and  encapsulated  in  this  Perspectives  book,  including  works  by 
John  Annett,  Frederick  Gilbreth,  Donald  Norman,  Jens  Rasmussen,  David  Woods,  the  classic  paper  on  "man- 
machine  task  analysis"  by  R.  B.  Miller,  and  others. 
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In  addition  to  providing  a  summary  at  the  end  of  each  chapter,  we  included  an  advance  organizer  at  the  beginning  of 
each  chapter.  These  organizers  take  the  form  of  "Concept  Maps"  that  layout  the  key  ideas.  As  concept  mapping 
spreads  around  the  world  in  its  various  applications  in  research,  education,  and  Web  work  (see  Canas  et  ai.,  2003; 
Canas  &  Novak,  2006;  Hoffman,  Coffey,  Novak,  &  Canas,  2005;  Hoffman  &  Lintern,  2006;  Novak,  1998),  we 
present  here  a  somewhat  novel  use  of  Concept  Maps-to  serve,  in  effect,  as  executive  summaries.  Learn  more  about 
CmapTools  (including  the  research  background)  and  get  a  free  download  at  http://www.ihmc.us.  A  detailed  "how¬ 
to"  discussion  of  the  use  of  Concept  Maps  as  a  knowledge  elicitation  method  for  creating  knowledge  models  is  in 
Crandall,  Klein,  and  Hoffman  (2006,  chap.  4). 

That  Was  Then 

Those  who  came  before  us  were  more  aware  of  the  key  issues  and  arguments  than  we  might  give  them  credit  for. 
Indeed,  many  modern  researchers  have  reinvented  the  wheel  and  given  it  new  names,  often  believing  their  work  is 
distinct  or  special.  In  fact,  Fernberger  (1937)  acknowledged  that  sporadic  amnesia  is  inevitable: 

Scientific  ideas  become  public  and  common  property  in  spite  of  anything  one  may  do  ....  Examples  of  particular 
processes  and  modes  of  explanation  become  current  in  a  laboratory  and  one  does  not  remember  whether  one  thought 
of  them  himself  or  whether  they  should  be  attributed  to  some  one  of  one's  colleagues,  (p.  xi) 

One  goal  of  this  book  is  to  draw  connections  from  past  approaches  to  current  methods,  tracing  the  historical  roots  of 
modern  Cognitive  Task  Analysis  (CTA),  tracing  the  pendulum  swings,  the  divergences  and  convergences.  We  hope 
that  for  a  next  generation  this  will  reduce  the  need  to  reinvent  the  wheel  and  encourage  efforts  to  improve  on  the 
wheels  we  already  have. 

Although  we  are  hesitant  to  overly  constrain  the  notion  of  CTA,  we  think  it  is  necessary  to  place  some  boundaries 
around  which  approaches  will  be  examined.  First  off,  we  need  to  look  at  the  meaning  of  the  word  task. 


Hoffman,  R.  R.,  Norman,  D.  O.,  &  Vagners,  J.  (2009).  “Complex  Sociotechnical  Joint  Cognitive  Work  Systems”?  . 
IEEE:  Intelligent  Systems,  82-89. 
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Summary  abstract: 

This  essay  continues  a  tradition  in  this  department:  deconstructing  the  meanings  of  various  buzz  phrases. ”1  It  calls 
out  a  cluster  of  phrases  that  use  the  word  system. 

Kinds  of  Systems?  Consider  the  following  six  systems-related  terms. 

Sociotechnical  systems  emerged  from  research  at  the  Tavistock  Institute  in  the  1950s  on  the  effects  of  the 
introduction  of  powered  machines  on  the  work,  management-labor  relations,  and  the  lives,  families,  and  societies  of 
coal  miners. 2 

Human-machine  systems  is  the  idea,  originating  in  industrial  psychology  in  the  World  War  I  era,  that  man  and 
machine  must  be  thought  of  as  a  single  system.  The  tipping  point  was  the  design  of  a  new  machine  tool’s  clutch, 
which  required  more  force  than  a  human  lathe  operator  could  muster.  The  engineers  had  to  learn  to  design  the 
human-machine  as  the  unit  of  analysis.  Researchers  developed  this  idea  further  during  World  War  II  in  work  on 
control  theory  and  cybernetics. 3  Use  of  the  term  system  expanded  after  around  1950.4 

Researchers  have  used  the  term  complex  systems  to  emphasize  the  importance  of  systems  theory  and  issues  of 
complexity  in  engineering  and  design. 5 

Human-system  integration  is  the  new  kid  on  the  block,  initially  a  refl  ection  of  the  US  Navy’s  interest  in  reducing 
the  crew  complement  on  ships.  More  recently,  the  concept  expanded  in  the  systems  engineering  community  to  refer 
not  only  to  the  traditional  goals  of  human  factors  such  as  safety,  trainability,  and  reliability  but  also  to  such  issues  as 
usability,  usefulness,  and  resilience  (the  key  concerns  of  cognitive  systems  engineering). 6 
Joint  cognitive  systems  describes  this  same  general  subject  matter,  with  “joint”  intended  to  focus  on  humans 
working  collaboratively  with  information  technology.  7 

Researchers  at  the  Institute  for  Human  and  Machine  Cognition  proposed  complex  cognitive  systems  as  the  name  of 
a  theory  based  on  the  idea  that  we  can  reinterpret  the  “principles”  of  human-  centered  computing  as  empirical 
nomological  generalizations — that  is,  scientifi  c  laws.  8 
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All  these  terms  relate  directly  and  immediately  to  human-centered  computing.  Their  point  is  to  create  technology 
and  work  methods  that  improve  work  processes  and  results,  while  also  improving  the  human  condition. 

Since  the  notion  of  systems  theory  became  coin  of  the  realm,  we’ve  added  many  mod  ifi  ers  onto  the  word 
“systems.”  For  example,  some  scholars  refer  to  “dynamical  systems,”  some  to  “large-scale  systems.”  Both  phrases 
are  thought  of  as  differing  from  run-of-the-mill  systems. 9  “Self-organizing”  systems  are  also  considered  more  than 
plain  old  systems,  in  that  they  increase  in  complexity  without  an  external  guiding  hand.  Entire  theories  are  built 
upon  the  notion  of  self-organization,  largely  (and  perhaps  not  surprisingly)  in  biological  systems.  10-12  Likewise, 
there  are  formidable  theoretical  scaffolds  discussing  “adaptive  systems.”  13 

Beyond  their  use  as  favored  terms  of  particular  disciplines  and  the  carriers  of  historical  baggage,  what  value  or 
meaning  do  we  add  by  prefixing  “system”  with  any  of  these  modifiers? 


Floffman,  R.  R.,  Roesler,  A.,  &  Moon,  B.  M.  (2004).  What  Is  Design  in  the  Context  of  Fluman-Centered 
Computing?  IEEE  Intelligent  Systems(July/August),  89-95. 
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Summary  abstract: 

Problem  solving  often  involves  recognizing  and  fiddling  with  tacit  assumptions.  Such  realization  can  often  come 
from  seeing  things  from  new  perspectives.  Appreciating  the  human-centered  perspective  may  offer  some  hope  for 
enriching  design’s  scientific  foundations  and  for  crafting  new  and  better  approaches  to  it.  Essays  in  this  department 
have  introduced  such  notions  as  the  Sacagawea  Principled 

Human-centered  computational  tools  need  to  support  active  organization  of  information,  active  search  for 
information,  active  exploration  of  information,  reflection  on  the  meaning  of  information,  and  evaluation  and  choice 
among  action  sequence  alternatives.  Certainly  this  suggests  a  constraint  on  or  a  goal  for  design,  but  how  do  we  go 
from  such  statements  to  actual  designs  that  accomplish  the  stated  goals? 

We  approach  this  class  of  question  by  considering  the  origins  of  and  historical  influences  on  the  notion  of  design, 
then  by  considering  the  assumptions  underlying  our  modern  conception  of  design  in  light  of  the  principles  of 
human-centered  computing. 


Hoffman,  R.  R.,  &  Shattuck,  L.  G.  (2006).  Should  We  Rethink  How  We  Do  OPORDs?  Military  Review,  March- 
April,  100-107. 
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Summary  abstract: 

The  U.S.  Army  has  used  a  structured  order  format  and  process  for  more  than  80  years.  The  1924  version  of  the  Field 
Service  Regulation  prescribed  “formatted  orders,  with  annexes,  maps,  and  tables. ”1  Staff  procedures  have  evolved 
since  then,  but  the  basic  structure  of  the  operation  order  (OPORD)  has  remained  essentially  the  same — five 
paragraphs  or  sections  that  describe  the  situation,  mission,  execution,  service  support,  and  command  and  signal.  The 
basic  process  for  creating,  sharing,  and  using  OPORDs  has  also  remained  essentially  the  same  and  is  “time- 
consuming  and  effortful.”  It  needs  to  be  revised. 2 

OPORDs  begin  when  a  higher  echelon  communicates  its  OPORD  to  a  lower  echelon.  The  lower  echelon 
commander  and  staff  review  the  OPORD  and  conduct  a  mission  analysis.  The  lower  level  commander  provides 
guidance  to  the  staff,  and  the  staff  enters  into  the  military  decisionmaking  process.  Typically,  the  staff  presents  the 
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commander  with  multiple  courses  of  action  (COAs),  the  commander  chooses  one  COA  and  expresses  his  intent,  the 
staff  creates  an  OPORD  and  passes  it  to  the  next  lower  echelon,  and  the  process  is  repeated  until  all  Soldiers  have 
been  told  what  is  expected  of  them.  OPORDs  are  also  passed  back  up  for  an  iterative  sweep  through  the  echelons 
and  final  approval. 

OPORDs  can  sometimes  be  hundreds  of  pages  long.  However,  clear,  concise  communication,  especially  of  the 
commander’s  intent,  is  a  critical  aspect  of  military  planning,  replanning,  and  operations.  The  effort  devoted  to 
training  in  writing  and  interpreting  OPORDs  in  the  Reserve  Officer  Training  Corps,  at  the  U.S.  Military  Academy, 
and  throughout  the  U.S.  Army  Training  and  Doctrine  Command  school  system  indicates  how  important  this  is. 3 
Having  recently  evaluated  OPORDs  and  conducted  empirical  studies  of  how  OPORDs  are  understood,  we  propose  a 
change  to  the  existing  OPORD  format  and  a  new  procedure  for  creating  OPORDs,  to  amplify  adaptive 
decisionmaking  at  all  echelons  and  improve  planning  for  joint  and  coalition  operations. 


Hoffman,  R.  R.,  &  Woods,  D.  D.  (2005).  Toward  a  Metatheory  for  the  Theory  of  Complex  Cognitive  Systems.  In 
IEEE. 
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Summary  abstract: 

Essays  in  this  department  have  presented  nine  propositions  that  we’ve  referred  to  as  principles  of  humancentered 
computing: 

•  The  Aretha  Franklin  Principle:  Do  not  devalue  the  human  in  order  to  justify  the  machine.  Do  not  criticize  the 
machine  in  order  to  rationalize  the  human.  Advocate  the  human-machine  system  in  order  to  amplify  both.l 

•  The  Sacagawea  Principle:  Human-centered  computational  tools  need  to  support  active  organization  of  information, 
active  search  for  information,  active  exploration  of  information,  reflection  on  the  meaning  of  information,  and 
evaluation  and  choice  among  action  sequence  alternatives.2 

•  The  Lewis  and  Clark  Principle:  The  human  user  of  the  guidance  needs  to  be  shown  the  guidance  in  a  way  that  is 
organized  in  terms  of  his  or  her  major  goals.  Information  needed  for  each  particular  goal  should  be  shown  in  a 
meaningful  form,  and  should  allow  the  human  to  directly  comprehend  the  major  decisions  associated  with  each 
goal. 2 

•  The  Envisioned  World  Principle:  The  introduction  of  new  technology,  including  appropriately  human-centered 
technology,  will  bring  about  changes  in  environmental  constraints  (that  is,  features  of  the  sociotechnical  system,  or 
the  context  of  practice).  Even  though  the  domain  constraints  might  remain  unchanged,  and  even  if  cognitive 
constraints  are  leveraged  and  amplified,  changes  to  the  environmental  constraints  might  be  negative. 3 

•  The  Fort  Knox  Principle:  The  knowledge  and  skills  of  proficient  workers  is  gold.  It  must  be  elicited  and  preserved, 
but  the  gold  must  not  simply  be  stored  and  safeguarded.  It  must  be  disseminated  and  utilized  within  the  organization 
when  needed.4 

•  The  Pleasure  Principle:  Good  tools  provide  a  feeling  of  direct  engagement.  They  simultaneously  provide  a  feeling 
of  flow  and  challenge. 5  •  The  Janus  Principle:  Human-centered  systems  do  not  force  a  separation  between  learning 
and  performance.  They  integrate  them.6 

•  The  Mirror-Mirror  Principle:  Every  participant  in  a  complex  cognitive  system  will  form  a  model  of  the  other 
participant  agents  as  well  as  a  model  of  the  controlled  process  and  its  environment.  7 

•  The  Moving  Target  Principle:  The  sociotechnical  workplace  is  constantly  changing,  and  constant  change  in 
environmental  constraints  might  entail  constant  change  in  cognitive  constraints,  even  if  domain  constraints  remain 
constant.  3 

The  term  “principle”  doesn’t  actually  do  much  work  in  science.  Colloquially,  it’s  used  as  a  tacit  reference  to  laws,  as 
in  “This  device  works  according  to  the  principle  of  gravity.”  What  are  these  so-called  principles?  Our  answer  leads 
to  additional  considerations  involving  the  use  of  the  principles. 


Hoffman,  R.  R.,  &  Yates,  J.  F.  (2005).  Decision(?)-Making(?).  IEEE:  Intelligent  Systems,  22-29. 
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Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Consider,  for  a  moment,  “What  are  we  making  intelligent  decision  aids  for?”  Computers,  including  intelligent 
systems,  assist  human  decision  making  in  many  ways.  1-3  Decision  aids  can  range  from  an  online  tool  provided  by 
Consumer  Reports  to  help  people  choose  a  refrigerator,  to  a  large  system  for  monitoring  an  industrial  process.  Group 
decision  aids  focus  on  supporting  communication. 4, 5  Expert  systems  can  be  considered  decision  aids,  as  can 
systems  that  use  statistical  methods  to  assist  in  diagnostic  procedures.  Decision  aids  can  provide  information 
involved  in  deciding  or  information  pertinent  to  evaluating  states  of  the  world.  To  support  these  situation  assessment 
and  monitoring  activities,  computers  can  integrate  and  display  information  and  assist  in  replanning.  6  When  we 
deconstruct  such  generic  tasks,  most  of  the  component  tasks  reduce  to  option  generation,  option  selection,  and 
outcome  evaluation.  7  For  these,  we  can  bring  to  bear  various  mathematical  techniques  such  as  utility  analysis.  8 
But  how  well  do  such  tools  really  help  humans  deal  with  the  difficulties  of  deciding  itself — for  instance,  adapting  to 
changing  circumstances  or  coping  with  situations  that  are  both  unfamiliar  and  infrequent?7  While  several  good 
analyses  of  decision-making  situations  and  situational  factors  exist  that  can  help  shape  decision-aid  architectures  and 
functionalities, 9, 10  has  anyone  done  a  corresponding  analysis  of  what  this  thing  called  the  decision  is?  After 
discussing  this  question,  we’ll  sharpen  the  focus  for  new  and  potentially  useful  applications  of  intelligent  systems 
technologies. 


Hoffman,  R.  R.,  Zachary,  W.,  Burns,  J.,  Drillings,  M.,  Hale,  C.  R.,  &  Linegang,  M.  (2008).  Panel  Discussion: 

Human  Total  Cost  of  Ownership:  Measuring  the  Impact  of  Human  Factors  on  System  Engineering.  In  In 
Proceedings  of  52nd  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  22-26  September  2008.  New 
York  City,  NY  USA.  New  York  City,  NY  USA:  HFES. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  procurement  process  often  results  in  information  systems  that  are  of  limited  usefulness,  usability  and 
understandability.  A  focus  on  short-term  cost  of  acquisition,  as  a  main  driving  force  in  procurement,  always  comes 
with  a  hefty  price  that  weighs  most  heavily  on  the  shoulders  of  those  who  have  to  conduct  cognitive  work  using  the 
new  technologies.  Procurement  that  is  driven  primarily  by  Designer-Centered  Design  and  the  goal  of  reducing 
immediate  cost  fails  because  it  does  not  recognize  the  value  of  the  human  component.  Zachary  et  al.  (2007) 
proposed  a  new  family  of  measures  for  use  in  procurement,  referred  to  collectively  as  Human  Total  Cost  of 
Ownership  (HTCO).  HTCO  might  be  defined  in  a  number  of  ways,  and  from  the  conceptual  definitions  one  might 
generate  a  number  of  operational  definitions  of  how  to  actually  calculate  metrics.  Panelists  will  address  the 
overarching  questions  of  HTCO  measures  and  their  integration  into  the  acquisition  and  development  process,  current 
obstacles  to  Human-Centered  Design,  ways  in  which  HTCO  might  gain  entry  into  the  procurement  process,  and 
alternative  approaches  to  creating  specific  HTCO  measurables. 


Hollnagel,  E.,  &  Woods,  D.  D.  (2005).  Joint  Cognitive  Systems:  Foundations  of  Cognitive  Systems  Engineering.: 
Taylor  &  Francis. 
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Summary  abstract: 

Deciding  what  to  write  in  a  preface  is  not  as  easy  as  it  seems.  If  there  is  anything  of  substance  to  say  about  the 
subject  matter,  it  should  clearly  be  in  the  main  text  of  the  book  rather  than  in  the  preface.  And  if  there  is  nothing  of 
substance  to  say,  then  why  say  it?  Prefaces  can  therefore  easily  become  the  place  where  the  authors  write  their  more 
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personal  views  on  matters  large  and  small,  express  their  gratitude  to  various  people,  and  lament  about  the  state  of  the 
world  in  general.  Prefaces  can  also  become  the  authors'  encomium  for  their  own  work.  Publishers  welcome  that, 
since  it  can  be  put  on  the  back  cover  or  on  the  website,  but  common  decency  usually  intervenes  in  time. 

After  this  beginning  there  are  pretty  few  options  left  for  us,  except  to  briefly  reminisce  on  how  this  book  came 
about.  The  collaboration  between  the  two  authors  began  around  1979  and  soon  led  to  the  ideas  that  became 
formulated  as  cognitive  systems  engineering  (CSE),  first  in  an  internal  report  in  1982,  and  later  as  a  paper  in  the 
International  Journal  of  Man-Machine  Studies.  About  ten  years  later,  in  the  beginning  of  the  1990s,  we  started 
talking  about  writing  a  book  on  CSE.  One  motivation  was  that  the  idea  seemed  to  have  caught  on;  another  was  that 
we  by  that  time  individually  and  together  had  produced  a  number  of  writings  that  both  had  developed  the  original 
ideas  further  and  illustrated  how  they  could  be  applied  in  practice. 

The  route  from  word  to  deed  is,  however,  usually  longer  than  initially  hoped  for.  While  the  intention  to  write  a  book 
together  was  never  abandoned,  progress  was  painfully  slow,  partly  because  we  both  had  entangled  ourselves  in  too 
much  interesting  work,  and  partly  because  the  much  coveted  opportunity  to  sit  down  together  for  a  couple  of  weeks 
on  a  desert  island  never  quite  materialised. 

In  the  end  the  book  became  a  reality  because  we  adopted  a  pragmatic  solution  -  in  good  according  with  the  ethos  of 
CSE.  Instead  of  writing  one  book  together,  the  project  was  split  into  two  parts,  leading  to  two  books  that 
complement  each  other.  While  we  appear  as  joint  authors  of  either  book,  the  order  differs  to  reflect  the  fact  that  the 
first  author  in  each  case  is  the  main  responsible  for  the  writing.  This  also  helps  to  solve  the  practical  problem  that  a 
single  book  would  have  been  rather  large.  Furthermore,  we  had  over  the  years  developed  different  styles  of  working 
and  writing,  one  being  more  contemplative,  the  other  being  more  practical. 

The  end  result  is  therefore  not  one  but  two  books  of  which  this  is  the  first.  Both  books  are  entitled  Joint  Cognitive 
Systems  with  the  subtitles  indicating  the  specific  focus  -  and  therefore  also  what  distinguishes  them  from  each  other. 
The  subtitle  of  the  present  book  is  'Foundations  of  Cognitive  Systems  Engineering',  while  the  subtitle  of  the  second 
book  is  'Patterns  in  Cognitive  Systems  Engineering'.  The  intention  is  that  either  book  can  be  read  independently,  but 
that  they  also  will  complement  each  other  by  emphasising  the  theoretical  and  the  practical  aspects  respectively.  It  is 
no  requirement  that  they  are  read  in  a  specific  order.  Yet  if  people  after  reading  either  one  would  find  it  necessary  to 
read  also  its  counterpart,  we  would  feel  we  had  achieved  our  purpose. 


Holzapfel,  H.,  Fuegen,  C.,  Denecke,  M.,  &  Waibel,  A.  (2003).  Integrating  Emotional  Cues  into  a  Framework  for 
Dialogue  Management.  In  In  Proceedings  of  the  Fourth  IEEE  International  Conference  on  Multimodal  Interfaces 
(ICMI’02).  Pittsburgh,  PA. 
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Summary  abstract: 

Emotions  are  very  important  in  human-human  communication  but  are  usually  ignored  in  human-computer 
interaction.  Recent  work  focuses  on  recognition  and  generation  of  emotions  as  well  as  emotion  driven  behavior.  Our 
work  focuses  on  the  use  of  emotions  in  dialogue  systems  that  can  be  used  with  speech  input  or  as  well  in  multi¬ 
modal  environments. 

This  paper  describes  a  framework  for  using  emotional  cues  in  a  dialogue  system  and  their  informational 
characterization.  We  describe  emotion  models  that  can  be  integrated  into  the  dialogue  system  and  can  be  used  in 
different  domains  and  tasks.  Our  application  of  the  dialogue  system  is  planned  to  model  multi-modal  human- 
computer-interaction  with  a  humanoid  robotic  system. 


Howard,  M.  V.,  &  Woods,  D.  D.  (2002).  The  'Squidgy':  An  Order  of  Magnitude  Display.  In  In  Proceedings  of 
Eleventh  European  Conference  on  Cognitive  Ergonomics  (ECCE-11).  Catania,  Sicily. 
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This  paper  presents  a  display  approach  aimed  at  visualizing  scales  and  ranges  that  span  many  orders  of  magnitude, 
with  a  focus  on  retaining  qualitative  aspects  of  non-linear  scales.  This  was  achieved  by  the  linking  of  linear  and  non¬ 
linear  frames  of  reference  and  allowing  the  user  to  explore  this  link  interactively  through  the  dynamic  reference  of 
the  representation.  The  approach  taken  is  one  which  is  theoretically  well  grounded  and  empirically  user  validated,  to 
provide  re-usable  design  seeds  for  future  development.  The  user  validation  was  targeted  at  eliciting  what 
‘usefulness’  means  for  such  a  representation  and  was  conducted  as  a  series  of  indepth,  qualitative  interviews 
focussing  future  situations  of  use  where  such  a  representation  may  be  available.  Results  show  that  the  key  aspects  of 
usefulness  are  the  tight  coupling  of  the  frames  of  reference,  high  dynamic  character  and  responsiveness  of  the 
interaction,  but  also  that  ‘usefulness’  has  both  a  cognitive  and  domain  component  which  interact  in  subtle  and 
complex  ways. 


Hughes,  T.,  Warwick,  W.,  Banks,  S.,  &  Garrambone,  M.  (2006).  Human  Behavior  Representations:  Reflections  on  a 
Decade  Past.  In  Presented  as  part  of  a  part  of  a  panel  discussion  for  the  Research  Development  and  Engineering 
Forum,  Fall  2006  Simulation  Interoperability  Workshop.  Orlando,  FL. 
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Summary  abstract: 

Panel  Charter 

OWhat  has  been  accomplished  over  the  past  decade? 

0Are  we  still  tracking  to  the  objectives  as  articulated  in  the  1995  M&S  Master  Plan? 

0Are  the  1995  objectives  still  valid? 

0What  challenges  remain? 


Hutchins,  S.,  Cosenzo,  K.  A.,  McDermott,  P.  L.,  Feng,  T.-d.,  Barnes,  M.,  &  Gacy,  M.  (2009).  An  Investigation  of 
the  Tactile  Communications  Channel  for  Robotic  Control.  In  In  Proceedings  of  53rd  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society,  San  Antonio,  TX. 
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ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  impacts  on  performance  of  three  different  forms  of  communication  (radio,  chat,  and  tactile  belt)  were  explored 
in  the  context  of  a  small  unmanned  ground  vehicle  (SUGV)  target  identification  task.  The  target  identification  task 
required  a  Commander  with  knowledge  of  target  locations  and  access  to  a  digital  map  displaying  the  current  SUGV 
position  and  orientation  to  direct  a  Soldier  remotely  operating  the  SUGV  to  the  targets  using  a  finite  set  of  eleven 
commands.  The  study  revealed  no  evidence  for  a  loss  of  soldier  performance  with  the  tactile  belt  communications 
channel.  The  finding  suggests  that  the  tactile  use  of  haptic  signals  may  be  feasible,  a  potentially  important  finding 
for  situations  requiring  covert  communications. 


Hutchins,  S.,  Josephson,  J.  R.,  Barnes,  M.,  Chandrasekaran,  B.,  McDermott,  P.  L.,  &  Miller,  T.  N.  (2009).  Chapter 
11.  An  Interactive  Decision  Support  Architecture  for  Visualizing  Robust  Solutions  in  High-Risk  Military 
Environments.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter: 
Foundations  and  Technology.  Section  II.  Presenting  Battlefield  Information  to  Warfighters:  48hrBooks  by  Alion 
Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving 
Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision 
Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board 
Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue 
SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 
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Summary  abstract: 

The  economic  climate  in  2009  is  an  important  reminder  of  the  crucial  role  that  risk  perception  and  risk 
misperception  play  in  decision  making  that  impacts  everyday  life.  Many  of  the  economic  decisions  such  as  percent 
of  acceptable  subprime  mortgages  were  actually  based  on  complex  statistical  models  using  overly  optimistic 
assumptions.  Military  decision  making  is  also  replete  with  examples  of  overconfidence  in  the  assumptions  of 
decisions,  i.e.,  assumptions  that  are  the  product  of  risk  perception  and/or  risk  misperception  by  otherwise  competent 
commanders  that  ultimately  result  in  disaster.  Examples  include  McArthur's  refusal  to  believe  the  Chinese  would 
attack  across  theYalu  River  during  the  Korean  conflict,  Montcalm's  decision  in  1759  not  to  defend  his  rear  flank 
against  Wolfe  because  of  the  supposedly  impregnable  cliffs  of  Quebec  City,  and  the  Maginot  Line  fiasco  in  World 
War  II. 

Our  research  focuses  on  instances  where  reality  does  not  match  the  assumptions  upon  which  decisions  were  made. 
We  developed  decision  tools  in  the  form  of  graphical  visualizations  and  human-computer  interaction  methods  that 
would  help  junior  commanders  understand  the  consequences  of  various  courses  of  action  (COAs).  Decisions  that 
result  in  successful  outcomes  under  a  variety  of  supportable  assumptions  operationally  define  our  concept  of 
robustness.  In  Chandrasekaran  (2008)  it  is  argued  that,  in  contrast  to  the  traditional  emphasis  on  optimality, 
planning  should  aim  to  produce  plans  that  are  robust,  i.e.,  plans  that  have  satisfactory  performance  even  when  reality 
fails  to  match  the  assumptions  and  models  used  in  planning.  Our  decision  tools  do  not  lead  to  a  single  optimal 
solution  but  rather  to  a  set  of  possible  solutions.  The  final  decision  is  left  to  the  commander  based  on  his/her 
instincts  and  weightings  of  various  military  criteria  (e.g.,  speed  versus  sensor  coverage).  Our  tools  are  designed  to 
impart  insight  rather  than  be  prescriptive  in  nature. 

Most  traditional  decision  research  approaches  model  uncertainty  in  the  world  probabilistically  and  offer  solutions 
that  are  adequate  on  average.  The  traditional  approaches  also  assume  a  defined  probability  space  and  a  rational 
actor.  Both  assumptions  can  be  misleading.  First,  much  of  the  uncertainty  in  the  world  cannot  be  modeled  in  a  fully 
defined  problem  space.  Second,  Kahneman  and  Tversky  (1979)  and  other  researchers  in  behavioral  economics 
conducted  numerous  studies  indicating  that  humans  do  not  necessarily  follow  normative  models.  In  general,  the 
results  indicate  humans  use  bounded  rational  solutions  or  heuristics  that  are  expedient  but  also  subject  to  bias.  In 
particular,  humans  often  misinteipret  and  misuse  probabilistic  estimates  related  to  risky  decisions.  The  point  is  not 
that  humans  make  poor  decisions;  rather,  they  usually  do  not  make  them  based  on  defined  probability  spaces.  The 
simple  probability  models  that  define  the  problem  spaces  may  be  inappropriate,  or  even  incomplete,  for  complex 
dynamic  systems.  In  many  combat  situations  outcome  probabilities  change  rapidly,  requiring  flexible  real-time 
replanning  as  the  situation  unfolds  rather  than  dependence  on  a  single  optimal  solution.  Robust  decisions  remain 
sufficient  even  when  reality  is  not  as  expected  because  robust  decisionmaking  acknowledges  that  models  ofthe 
world  are  incomplete  and  human  decision  making  is  error-prone. 

The  research  described  is  an  attempt  to  understand  the  interplay  between  robust  solutions,  normative  models,  and 
the  ability  of  the  military  planner  to  understand  the  risk  environment.  Our  research  assumes  the  commander  will 
base  his/her  decisions  on  multiple  cues  with  relative  importance  that  will  change  as  the  combat  environment 
changes.  Based  on  that  assumption,  we  anticipate  commanders  will  develop  more  robust  solutions  if  they  are  able  to 
understand  various  decision  outcomes  and  their  consequences.  We  additionally  expect  that  planning  time  will  be 
reduced  with  tools  that  allow  the  operator  to  filter  unreasonable  solutions,  circumventing  the  computational 
explosion  inherent  in  any  decision  tree. 

The  uncertainty  of  the  real  world  poses  a  variety  of  challenges  in  the  design  of  decision  support  systems  (055). 
Decision  support  systems  attempt  to  mediate  uncertainty  through  decision  tools  such,  such  as  those  examined,  here 
toward  robust  decisions.  Contingency  planning  against  the  most  likely  events  or  intentions  is  sometimes  appropriate 
for  dealing  with  uncertain  situations.  Other  times,  uncertainty  may  be  addressed  by  simply  exploring  the  robustness 
of  selected  plans  and  understanding  how  the  plan  may  perform  if  reality  does  not  quite  match  the  planning 
assumptions.  For  example,  a  robust  plan  would  provide  good  coverage  even  when  the  actual  enemy  avenue  of 
approach  turns  out  to  be  different  from  that  assumed  for  planning.  A  plan  is  also  robust  if  it  provides  good  coverage 
when  some  of  the  sensors  fail,  either  en  route  or  during  surveillance.  The  risks  and  uncertainties  need  to  be 
presented  to  the  planner  in  a  way  that  is  meaningful  and  protective  against  biases.  While  decision  support  is  often 
couched  in  terms  of  making  an  optimal  decision,  helping  the  decision  maker  understand  the  decision  space  (i.e., 
develop  a  sense  of  relations  between  decision  variables  and  outcomes  in  different  parts  of  the  decision  space)  is 
equally,  if  not  more  important.  If  after  the  selected  plan  is  implemented  external  events  call  for  rapid  replanning, 
then  the  planner  can  exploit  his/her  understanding  of  the  decision  space  to  rapidly  and  effectively  respond. 

To  investigate  the  concept  of  robustness  for  risk  management,  we  chose  a  sensor  allocation  problem.  In  addition  to 
being  a  vehicle  for  exploring  robust  decision  making,  the  problem  is  also  intrinsically  important  militarily  and 
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relates  to  other  problems  such  as  persistent  surveillance,  border  incursions,  and  force  protection.  The  sensor 
allocation  problem  is  a  good  application  of  one  of  our  research  goals:  how  to  present  interactive  information  to 
planners  that  allows  them  to  understand  the  consequences  of  various  courses  of  action  while  maintaining  flexible 
options  in  cases  of  sudden  combat  reversals. 


Hutchins,  S.,  McDermott,  P.  L.,  &  Barnes,  M.  (2008).  Frame  Consistency  in  Multi-Attribute  Risk  Preference 
Decisions.  In  In  Proceedings  of  52nd  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  22-26 
September  2008.  New  York  City,  NY  USA.  New  York  City,  NY  USA:  HFES. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Resource  allocation  under  risk  and  uncertainty  is  a  complex  problem,  specifically  when  making  course  of  action 
planning  decisions  on  the  battlefield.  Previous  research  in  this  domain  has  demonstrated  the  performance  impact  of 
visualization  aids,  feedback,  practice,  information  frame,  and  time  pressure.  However,  the  focus  of  prior  research 
has  been  on  how  to  help  operators  determine  where  to  optimally  place  a  single  asset.  The  current  research  addressed 
the  same  domain,  but  with  the  multi-attribute  decision  of  how  to  best  allocate  a  suite  of  robotic  assets  across  the 
battlefield.  We  explored  the  impact  of  information  frame  and  time  pressure  on  decision  performance  and  situation 
awareness,  but  further  manipulated  whether  the  information  frame  of  decision  criteria  were  forced  to  a  single 
consistent  frame  or  allowed  to  vary  according  to  which  term  was  intuitive,  regardless  of  frame.  We  found  both  faster 
response  times  and  more  accurate  responses  within  consistently-framed  displays  over  mixed-frame  displays.  In 
addition,  response  time  varied  less  in  the  consistently-framed  displays  than  the  mixed-frame  displays. 


Hutton,  R.,  Stanard,  T.,  Warwick,  W.,  &  Mcllwaine,  S.  (2003).  Computationally  Modeling  Recognition-Primed 
Decisions:  Conceptual  Insights.  In  In  Proceedings  of  Fairborn,  OH  CTA  Conference:  Klein  Associates  Inc. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
•RPD  Primer 

•High  level  phenomenon  vs.  processes  and  mechanisms 
•What  is  a  decision? 

•What  is  recognition? 

•Expert/novice  differences  in  RPD 

•What  are  expectancies?  How  do  they  relate  to  mental  models  and  mental  simulation? 

•Do  simulation  environments  support  perceptual/cognitive  activities  of  simulated  agents? 

Summary  abstract: 

RPD  Primer 

•Describes  how  experts  make  good  decisions  in  tough  situations*Emphasis  on  situation  assessment 
•Emphasis  on  experience  and  recognition  based  on  typicality 
•Emphasizes  decision  cycles  vs.  choice  points 
•Emphasis  on  satisficing 

•Emphasis  on  mental  simulation  of  plausible  options  (not  concurrent  evaluation  of  candidate  courses  of  action 


Hutton,  R.  J.  B.,  Miller,  T.  E.,  &  Thordsen,  M.  L.  (2003).  Chapter  17.  Decision  Centered  Design:  Leveraging 
Cognitive  Task  Analysis  in  Design.  In  E.  Hollnagel  (Ed.),  Handbook  of  Cognitive  Task  Design  (pp.  383-416).  10 
Industrial  Avenue,  Mahwah,  New  Jersey  07430:  Lawrence  Erlbaum  Associates,  Inc.  Also  In  Proceedings  of  XVth 
Triennial  Congress  of  the  International  Ergonomics  Association.  Seoul,  Korea. 


Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
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Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Decision-centered  design  (DCD)  is  a  design  approach  that  focuses  on  understanding  and  supporting  cognitive  tasks. 
Research  in  the  areas  of  naturalistic  decision  making  and  expertise  has  provided  the  cognitive  models  and  analytic 
methods  that  enable  this  approach.  DCD  provides  a  means  for  communication  and  understanding  between  designers 
and  the  individuals  for  whom  interventions  are  being  designed.  It  targets  the  critical,  often  challenging  cognitive 
tasks  confronting  these  individuals,  including  individual  tasks  in  which  cognitive  performance  may  break  down  and 
teamlevel  tasks  that  require  collaboration  and  coordination.  The  approach  consists  of  task  analysis,  design,  and 
evaluation. 


Iyer,  N.  S.,  &  Josephson,  J.  R.  (2001).  Multicriterially  Best  Explanations.  In  K.P.  Jantke  and  A.  Shinohara  (Eds.), 
Discovery  Science,  Lecture  Notes  in  Artificial  Intelligence  2226,  Pp.  128-140.  Also  in  Proceedings  of  Discovery 
Science  2001:  The  4th  International  Conference  on  Discovery  Science.  Washington  Dc,  2002.  New  York,  NY: 
Springer  Verlag.  ISBN  3-540-42956-5. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Inference  to  the  best  explanation,  IBE,  (or  abduction)  requires  finding  the  best  explanatory  hypothesis,  from  a  set  of 
rival  hypotheses,  to  explain  a  collection  of  data.  The  notion  of  best,  however,  is  multicriterial  and  the  available  rival 
hypotheses  might  be  variously  good  according  to  different  criteria.  Thus,  one  can  view  the  abduction  problem  as  that 
of  choosing  the  best  hypothesis  from  among  a  set  of  multicriterially  evaluated  hypotheses  -  i.e  as  a  multiple  criteria 
decision  making  problem  (MCDM).  In  the  absence  of  a  single  hypothesis  that  is  the  best  along  all  dimensions  of 
goodness,  the  MCDM  problem  becomes  especially  hard.  The  Seeker-Filter-Viewer  architecture  provides  an 
effective  and  natural  way  to  use  computer  power  to  assist  humans  to  solve  certain  classes  of  MCDM  problems.  In 
this  paper,  we  apply  an  MCDM  perspective  to  the  abductive  problem  of  red-cell  antibody  identification  and  present 
the  results  obtained  by  using  the  S-F-V  architecture. 


Jefferson,  T.,  Jr.,  Ferzandi,  L.,  &  McNeese,  M.  (2004).  Assessing  the  Impact  of  Hidden  Knowledge  Profiles  on 
Distributed  Cognition  and  Team  Decision-Making:  Recounting  the  Development  of  the  Neocities  Simulation.  In  In 
Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  study  continues  research  from  McNeese  (2000)  and  was  designed  to  examine  the  effects  of  hidden  knowledge 
profiles  on  perceptually  anchored  team  cognition  and  knowledge  transfer  in  distributed  teams.  Previous  work 
showed  that  individuals  in  perceptually  anchored  distributed  teams  were  able  to  quickly  access  applicable 
knowledge,  then  transfer  that  knowledge  to  answer  similarly  situated  problems.  Perceptual  anchors  provide  the  basis 
for  formulating  team  mental  models,  which  can  be  used  to  assess  situations  and  resolve  differences  in  individual, 
unique  knowledge.  In  the  present  experiment,  it  was  again  hypothesized  that  individuals  working  in  perceptually 
anchored  distributed  teams  would  be  better  able  to  transfer  knowledge,  than  teams  without  anchors.  It  was  also 
hypothesized  that  perceptually  anchored  distributed  teams  would  be  better  able  to  share  uniquely  held  information, 
make  better  decisions,  identify  information  discrepancies,  and  overcome  the  presence  of  hidden  knowledge  profiles 
better  than  non-perceptually  anchored  teams.  Preliminary  findings  are  discussed. 
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Jin,  J.,  Rothrock,  L.,  McDermott,  P.  L.,  &  Barnes,  M.  (2009).  Using  the  Analytic  Hierarchy  Process  to  Examine 
Judgment  Consistency  in  a  Complex  Multi-Attribute  Task.  IEEE  Transactions  on  Systems  Man  &  Cybernetics,  Part 
A  Systems  and  Humans. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Analytic  Hierarchy  Process  (AHP),  framing  effect,  judgment  consistency,  multiattribute  task. 

Summary  abstract: 

This  paper  investigates  the  impact  of  framing  and  time  pressure  on  human  judgment  performance  in  a  complex 
multiattribute  judgment  task.  We  focus  on  the  decision  process  of  human  participants  who  must  choose  between 
pairwise  alternatives  in  a  resource-allocation  task.  We  used  the  Analytic  Hierarchy  Process  (AHP)  to  calculate  the 
relative  weights  of  the  four  alternatives  (i.e.,  Cl,  C2,  C3,  and  C4)  and  the  judgment  consistency.  Using  the  AHP,  we 
examined  two  sets  of  hypotheses  that  address  the  impact  of  task  conditions  on  the  weight  prioritization  of  choice 
alternatives  and  the  internal  consistency  of  the  judgment  behavior  under  varying  task  conditions.  The  experiment 
simulated  the  allocation  of  robotic  assets  across  the  battlefield  to  collect  data  about  an  enemy.  Participants  had  to 
make  a  judgment  about  which  asset  to  allocate  to  a  new  area  by  taking  into  account  three  criteria  related  to  the 
likelihood  of  success.  We  manipulated  the  information  frame  and  the  nature  of  the  task.  We  found  that,  in  general, 
participants  gave  significantly  different  weights  to  the  same  alternatives  under  different  frames  and  task  conditions. 
Specifically,  in  terms  of  ln-transformed  priority  weights,  participants  gave  significantly  lower  weights  to  C2  and  C4 
and  higher  weight  to  C3  under  gain  frame  than  under  loss  frame,  and  also,  under  different  task  conditions  (i.e., 

Tasks  #1,  #2,  and  #3),  participants  gave  significantly  higher  weight  to  C4  in  Task  #1,  lower  weights  to  Cl  and  C4, 
higher  weight  to  C3  in  Task  #2,  and  lower  weight  to  C3  in  Task  #3.  Furthermore,  we  found  that  the  internal 
consistency  of  the  decision  behavior  was  worse,  first,  in  the  loss  frame  than  the  gain  frame  and,  second,  under  time 
pressure.  Our  methodology  complements  utility-theoretic  frameworks  by  assessing  judgment  consistency  without 
requiring  the  use  of  task-performance  outcomes.  This  work  is  a  step  toward  establishing  a  coherence  criterion  to 
investigate  judgment  under  naturalistic  conditions.  The  results  will  be  useful  for  the  design  of  multiattribute 
interfaces  and  decision  aiding  tools  for  real-time  judgments  in  time-pressured  task  environments. 


Johnson,  M.,  Feltovich,  P.  J.,  Bradshaw,  J.  M.  &  Bunch,  L.  (2008).  Human-Robot  Coordination  through  Dynamic 
Regulation.  In  In  Proceedings  of  2008  IEEE  International  Conference  on  Robotics  and  Automation  (ICRA-08), 
Pasadena,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Several  key  aspects  of  coordination  such  as  teamwork,  roles,  and  communication,  are  enabled  and  driven  by,  and 
even  largely  defined  by,  various  systems  of  regulation.  One  key  feature  of  all  these  elements  in  human  coordination 
is  their  dynamic  nature.  We  have  developed  a  framework  to  provide  a  dynamic  regulatory  system  for  supporting 
coordination  in  human-robot  teamwork.  This  framework  supports  the  definition  and  functions  of  roles  within  teams, 
as  well  as  the  creation  of  subteams  and  the  roles  within  them.  It  also  serves  to  regulate  communications  in  support  of 
coordination.  We  have  demonstrated  our  system  with  a  team  of  two  humans  and  five  robots  performing  advanced 
coordination  while  trying  to  apprehend  an  intruder  hiding  on  a  cluttered  pier.  This  work  lays  the  foundation  for 
human-robot  coordination  based  on  dynamic  regulation. 


Johnson,  M.,  Bradshaw,  J.  M.,  Jung,  H.,  Suri,  N.  &  Carvalho,  M.  (2008).  Policy  Management  across  Multiple 
Platforms  and  Application  Domains.  In  In  Proceedings  of  2008  IEEE  Conference  on  Policy,  Palisades,  NY. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
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Summary  abstract: 

One  of  the  challenges  of  building  a  policy  management  framework  is  making  it  flexible  enough  to  handle 
differences  in  both  policy  semantics  and  enforcement  strategies  across  multiple  platforms  and  application  domains. 
The  system  must  be  expressive  enough  in  each  application  domain  to  provide  the  richness  needed  for  interesting 
policies.  It  must  also  provide  a  simple  and  flexible  enforcement  mechanism  for  adaptation  to  a  variety  of  systems.  In 
this  paper  we  discuss  the  application  of  the  KAoS  policy  services  framework  to  human-robot  teamwork — an 
application  that  involves  a  variety  of  application  domains  and  enforcement  at  different  levels  of  control;  from  low 
level  network  resource  control  to  high  level  organizational  constraints  and  coordination  management.  The  study 
culminated  in  an  outdoor  field  exercise  that  required  coordination  of  mixed  sub  teams  composed  of  two  people  and 
five  robots  whose  task  was  to  find  and  apprehend  an  intruder  on  a  Navy  pier. 


Johnson,  M.,  Bradshaw,  J.,  Feltovich,  P.,  Jeffers,  R.,  Jung,  H.,  &  Uszok,  A.  (2006).  A  Semantically-Rich  Policy- 
Based  Approach  to  Robot  Control.  In  In  Proceedings  of  Third  International  Conference  on  Informatics  in  Control, 
Automation,  and  Robotics  (ICINCO  2006),  Setfbal,  Portugal. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Policy,  semantic,  authorization,  obligation,  ontology 

Summary  abstract: 

In  this  paper  we  describe  our  approach  to  enhancing  control  of  robotic  systems  by  providing  domain  and  policy 
services  via  KAoS.  Recently  developed  languages  such  as  OWL  provide  a  powerful  descriptive  logic  foundation 
that  can  be  used  to  express  semantically  rich  relationships  between  entities  and  actions,  and  thus  create  complex 
context  sensitive  policies.  KAoS  provides  a  tool  to  create  policies  using  OWL  and  an  infrastructure  to  enforce  these 
policies  on  robots.  We  contend  that  a  policy-based  approach  can  provide  significant  advantages  in  controlling 
robotic  systems  and  is  a  much  more  natural  way  for  operators  to  interact  with  and  manage  multiple  robots. 


Johnson,  M.,  Bradshaw,  J.  M.,  Feltovich,  P.  J.,  Jonker,  C.  M.,  &  Riemsdijk,  B.  v.  (2009).  Coactive  Design.  Why 
Interdependence  Must  Shape  Autonomy.  In  In  Proceedings  of  International  Conference  on  Engineering  Societies  in 
the  Agent's  World  (ESAW  '09),  Utrecht,  The  Netherlands.  Human-Agent-Robot  Teamwork  Workshop  (HART). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Categories  and  Subject  Descriptors 
1.2.9  [Robotics]:  Autonomous  vehicles 
General  Terms 

Design,  Human  Factors,  Theory. 

Keywords 

Coactive,  autonomy,  interdependence,  joint  activity. 

Summary  abstract: 

This  paper  introduces  Coactive  Design  as  a  new  approach  to  address  the  increasingly  sophisticated  roles  for  people 
and  agents  in  mixed  human-agent  systems.  The  basic  premise  of  Coactive  Design  is  that  the  underlying 
interdependence  of  joint  activity  is  a  critical  design  feature.  When  designing  the  capabilities  that  make  an  agent 
autonomous,  the  process  should  be  guided  by  an  understanding  of  the  interdependence  in  the  joint  activity.  This 
understanding  is  then  used  to  shape  the  operation  of  agent  capabilities  and  enable  appropriate  interaction.  The 
success  of  future  human-agent  teams  hinges  not  merely  on  trying  to  make  agents  more  autonomous,  but  also  in 
striving  to  make  them  more  capable  of  sophisticated  interdependence 


Johnson,  M.,  Chang,  P.,  Jeffers,  R.,  Bradshaw,  J.,  Breedy,  M„  Bunch,  L.,  Kulkami,  S.,  Lott,  J.,  Suri,  N.,  Uszok,  A., 
&  Soo,  V.-W.  (2003).  KAoS  Semantic  Policy  and  Domain  Services:  An  Application  of  DAML  to  Web  Services- 


184 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 

Based  Grid  Architectures.  In  In  Proceedings  of  AAMAS  03  Workshop  on  Web  Services  and  Agent-Based 
Engineering.  Melbourne,  Australia. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Categories  and  Subject  Descriptors 

1.2.11  [Artificial  Intelligence]  Distributed  Artificial  Intelligence  -  Intelligent  agents,  Multiagent  systems 
1.2.4  [Artificial  Intelligence]  Knowledge  Representation  Formalisms  and  Methods  -  Representation  languages, 
Semantic  networks 

1.2.3  [Artificial  Intelligence]  Deduction  and  Theorem  Proving  -  Inference  engines 
General  Terms 

Management,  Security,  Standardization,  Languages 
Keywords 

policy,  domain,  access  control,  authorization,  KAoS.  Grid  Computing,  Web  Services,  DAML,  OWL,  Semantic 
Web,  Semantic  Grid,  description  logic,  software  agent 

Summary  abstract: 

This  paper  introduces  a  version  of  KAoS  Semantic  Policy  and  Domain  Services  that  has  been  developed  to  support 
Web  Services-based  (i.e.,  OGSA-compliant)  Grid  Computing  Architectures.  While  initially  oriented  to  the  dynamic 
and  complex  requirements  of  software  agent  applications,  KAoS  services  are  now  being  extended  to  work  equally 
well  with  both  agent  and  non-agent  clients  on  a  variety  of  more  general  distributed  computing  platforms.  The 
OGSA-compliant  version  of  KAoS  services  allows  fine-grained  policy-based  management  of  registered  Grid 
services  as  well  as  opening  additional  opportunities  for  the  use  of  agents  on  the  grid. 


Johnson,  M.,  Feltovich,  P.  J.,  &  Bradshaw,  J.  M.  (2008).  R2  Where  Are  You?  Designing  Robots  for  Teamwork  with 
Humans.  In  In  Proceedings  of  ICRA  2008  Workshop  on  Social  Interaction  with  Intelligent  Indoor  Robots  (SI3R 
2008),  Pasadena,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  majority  of  robotic  systems  today  are  designed  by  first  building  a  robot  that  can  perform  some  tasks,  adding  an 
interface,  and  then  tiying  to  figure  out  why  the  interaction  is  unnatural  and  the  collaboration  is  non-existent. 
Collaboration  must  be  designed  into  the  system  from  the  start.  There  has  been  a  lot  of  work  on  both  the  interface  and 
the  autonomy  ends  of  such  systems,  but  the  critical  component  to  facilitate  coordination  lies  in  the  middle  and  has 
had  only  limited  attention  in  the  robotics  world.  This  collaborative  middle  layer  should  drive  the  design  of  both 
interface  and  autonomy.  In  this  paper  we  will  provide  a  detailed  description  of  the  type  of  collaboration  envisioned 
and  the  characteristics  associated  with  this  type  of  joint  activity.  These  will  be  used  to  establish  the  design 
requirements  for  collaborative  humanrobot  systems  and  present  a  cohesive  portrait  of  the  essential  components 
necessary  to  achieve  the  ideal  goal.  In  addition,  some  critical  challenge  areas  are  highlighted. 


Johnson,  M.,  Kulkami,  S.,  Raj,  A.,  Carff,  R.,  &  Bradshaw,  J.  M.  (2005).  Ami:  An  Adaptive  Multi-Agent  Framework 
for  Augmented  Cognition.  In  In  Proceedings  of  1st  International  Conference  on  Augmented  Cognition.  Mahwah, 

NJ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  this  paper,  we  report  on  our  efforts  to  develop  an  Adaptive  Multi-agent  Integration  framework,  called  AMI,  and 
its  application  in  the  development  of  Augmented  Cognition  (AugCog)  systems.  Key  components  of  AMI  include:  1) 
Pluggable  sensor  architecture  with  dynamically  substitutable  measures  for  flexible,  optimal  exploitation  of  all 
available  multi-sensory  channels;  and  2)  An  industrial-strength  integration  approach  and  providing  high 
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faulttolerance  and  dynamic  workload  distribution  across  virtually  any  implementation  platform.  AMI  leverages 
IHMC’s  Knowledgeable  Agent-oriented  Systems  (KAoS)  services  to  provide  a  flexible  and  generic  approach  for 
linking  components.  For  AugCog  systems,  these  components  can  be  for  sensing,  detecting  cognitive  state,  and 
modulating  the  application  itself.  AMI  has  served  as  the  integration  platform  for  a  large  multi-institutional  team  led 
by  Honeywell  Laboratories  under  funding  from  DARPA’s  Improving  Warfighter  Information  Intake  Under  Stress 
program.  Multiple  sensors  and  cognitive  state  measurement  gauges  have  been  independently  developed  by  members 
of  the  team  and  straightforwardly  integrated  for  rapid  prototype  development  and  evaluation.  The  flexibility  of  the 
AMI  framework  has  been  successfully  demonstrated  by  accommodating  the  myriad  of  changes  to  hardware  and 
algorithms  required  by  the  research  advances  achieved  throughout  the  project. 


Johnson,  M.  J.,  Intlekofer,  K.,  Jr.,  Jung,  H.,  Bradshaw,  J.  M.,  James  Allen,  Suri,  N.,  &  Carvalho,  M.  (2008). 
Coordinated  Operations  in  Mixed  Teams  of  Humans  and  Robots.  In  In  Proceedings  of  2008  IEEE  International 
Conference  on  Distributed  Human-Machine  Systems  (DHMS  2008).  Athens,  Greece. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Robot  and  Human  Cooperation,  Team  Formation,  Teamwork  policies,  Agile  Computing,  Dialogue-Based 
Collaboration. 

Summary  abstract: 

The  goal  of  this  research  was  to  facilitate  the  formation  of  mixed  teams  of  humans  and  robots  that  could  perform 
complex  tasks  in  the  real  world.  We  developed  a  complete  end-to-end  system  to  meet  this  goal.  We  used  a  natural 
language  based  multi-modal  interface  to  enable  simple  interaction  for  the  people  on  the  team.  We  used  policy 
regulated  behavior  for  the  robots  to  ensure  effective  coordination  from  the  robotic  teammates.  We  also  employed  an 
advanced  network  infrastructure  to  enable  robust  and  reliable  communications  among  team  members. 


Jones,  D.  G.,  Bolstad,  C.  A.,  Riley,  J.  M.,  Endsley,  M.  R.,  &  Shattuck,  L.  G.  (2003).  Situation  Awareness 
Requirements  for  the  Future  Objective  Force.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference 
2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of 
Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

This  paper  demonstrates  how  a  Goal  Directed  Task  Analysis  (GDTA)  can  be  used  to  provide  insight  into  potential 
allocations  of  staff  functions  within  the  Objective  Force  Units  of  Action.  The  GDTA  is  a  methodology  that 
delineates  the  goals,  decisions,  and  SA  requirements  essential  for  achieving  a  particular  objective.  In  evaluating  the 
responsibilities  assigned  to  different  components  of  a  proposed  command  structure,  the  GDTA  can  provide  support 
for  three  types  of  assessments:  (1)  an  assessment  of  essential  information  requirements,  (2)  an  assessment  of  likely 
cognitive  workload  associated  with  the  proposed  structure,  and  (3)  an  assessment  of  potential  problems  and/or 
benefits  associated  with  a  particular  allocation  of  responsibilities.  This  type  of  assessment  will  highlight  areas  where 
soldiers’  information  needs  are  not  being  adequately  met,  where  particular  staff  function  allocations  are  overly 
demanding  on  the  soldiers’  cognitive  resources,  and  where  particular  combinations  of  responsibilities  within  and 
across  cells  can  create  unexpected  consequences  that  negatively  impact  the  soldiers  ability  to  successfully  complete 
their  mission. 


Jones,  D.  G.,  Quoetone,  E.  M.,  Ferree,  J.  T.,  Magsig,  M.  A.,  &  Bunting,  W.  F.  (2003).  An  Intial  Investigation  into 
the  Cognitive  Processes  Undelying  Mental  Projection.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 
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Summary  abstract: 

The  objective  of  this  study  is  to  gain  insight  into  the  cognitive  mechanisms  that  enable  a  person  to  project  how  a 
system  or  environment  will  change  over  time.  Two  potential  mechanisms  for  mental  projection  are  examined:  (1) 
mental  simulation  (performing  a  mental  play  by  play)  and  (2)  pattern  matching  (identifying  critical  features  of  the 
situation  and  matching  those  with  previous  experience).  A  study  was  performed  in  cooperation  with  the  National 
Weather  Service  Warning  Decision  Training  Branch  in  which  28  severe  weather  warning  forecasters  participated  in 
an  interval-based  simulation.  At  three  decision  points  in  this  simulation,  participants  were  asked  to  assess  the  threat 
level  for  tornado,  wind,  hail,  and  flash  flood  and  to  delineate  their  rationale  for  each  assessment.  Results  are 
presented  that  support  the  existence  of  pattern  matching  and  mental  simulation  strategies,  that  suggest  different 
strategies  may  be  effective  in  different  situations,  and  that  indicate  forecasters  utilize  both  strategies. 


Jones,  L.  A.,  Nakamura,  M.,  &  Lockyer,  B.  (2004).  Development  of  a  Tactile  Vest.  In  In  Proceedings  of  IEEE 
Computer  Society  12th  International  Symposium  on  Haptic  Interfaces  for  Virtual  Environments  and  Teleoperator 
Systems,  pp.  82-89. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  research  is  focused  on  the  development  of  a  torsobased  haptic  display  that  can  be  used  to  present  navigational 
cues  to  a  human  operator.  The  requirements  for  this  display  are  that  it  can  be  worn  on  a  mobile  operator,  is  light 
weight  and  robust.  Two  tactile  vests  are  being  developed  to  meet  these  objectives,  one  based  on  small  electric 
motors  and  the  other  on  contractile  shapememory  alloy  (SMA)  fibers.  Four  electromechanical  actuators  were 
evaluated  for  use  in  the  vest  and  a  small  vibration  motor  was  selected  for  the  initial  prototype.  A  3x3  array  was 
fabricated  for  psychophysical  testing  and  positioned  on  the  lower  back.  High  accuracy  in  perceiving  the  direction  of 
tactor  activation  was  achieved  by  all  subjects  in  this  task.  A  factor  array  based  on  SMA  actuators  was  fabricated  and 
tested  and  results  indicate  that  this  low-bandwidth  device  can  generate  tactile  inputs  on  the  torso  that  are  perceptible. 


Jones,  L.  A.,  &  Sarter,  N.  (2008).  Tactile  Displays:  Guidance  for  Their  Design  and  Application.  Human  Factors,  50, 
90-111. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

This  article  provides  an  overview  of  tactile  displays.  Its  goal  is  to  assist  human  factors  practitioners  in  deciding 
when  and  how  to  employ  the  sense  of  touch  for  the  purpose  of  information  representation.  The  article  also  identifies 
important  research  needs  in  this  area.  Background:  First  attempts  to  utilize  the  sense  of  touch  as  a  medium  for 
communication  date  back  to  the  late  1950s.  For  the  next  35  years  progress  in  this  area  was  relatively  slow,  but  recent 
years  have  seen  a  surge  in  the  interest  and  development  of  tactile  displays  and  the  integration  of  tactile  signals  in 
multimodal  interfaces.  Athorough  understanding  of  the  properties  of  this  sensory  channel  and  its  interaction  with 
other  modalities  is  needed  to  ensure  the  effective  and  robust  use  of  tactile  displays.  Methods:  First,  an  overview  of 
vibrotactile  perception  is  provided.  Next,  the  design  of  tactile  displays  is  discussed  with  respect  to  available 
technologies.  The  potential  benefit  of  including  tactile  cues  in  multimodal  interfaces  is  discussed.  Finally,  research 
needs  in  the  area  of  tactile  information  presentation  are  highlighted.  Results:  This  review  provides  human  factors 
researchers  and  interface  designers  with  the  requisite  knowledge  for  creating  effective  tactile  interfaces.  It  describes 
both  potential  benefits  and  limitations  of  this  approach  to  information  presentation.  Conclusion:  The  sense  of  touch 
represents  a  promising  means  of  supporting  communication  and  coordination  in  human-human  and  human-machine 
systems.  Application:  Tactile  interfaces  can  support  numerous  functions,  including  spatial  orientation  and  guidance, 
attention  management,  and  sensory  substitution,  in  a  wide  range  of  domains. 
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Jones,  L.  A.  (2009).  Chapter  8.  Communication  Via  the  Skin:  The  Challenge  of  Tactile  Displays.  In  P.  McDermott 
and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section 
II.  Presenting  Battlefield  Information  to  Warfighters:  48hrBooks  by  Alion  Science  &  Technology  Corporation 
under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and 
Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative 
Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the 
Soldier  Decision  Maker.  July  22-23,  2009.  1100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003. 
http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

Tactile  communication  systems  represent  a  promising  technology  that  can  be  used  to  present  information  to  soldiers 
in  a  variety  of  contexts  by  utilizing  a  relativelyundemsed  sensory  channel  to  convey  information  that  is  both  private 
and  discreet.  Applications  of  this  technology  include  delivering  vibrotactile  cues  to  soldiers  to  assist  in  navigation  or 
threat  location  in  the  battlefield,  providing  tactile  feedback  to  increase  situational  awareness  in  virtual  environments 
used  for  training,  and  employing  tactile  cues  to  enhance  the  representation  of  information  in  multi-sensory  displays 
used  for  planning  and  decision  making.  The  opportunity  to  use  the  sense  of  touch  in  these  diverse  application 
domains  arose  as  a  result  of  tactile  display  technologies  becoming  more  sophisticated  due  to  the  reduced  power 
requirements  of  actuators  used  in  the  displays,  and  the  option  of  wireless  communication  for  mobile  users.  This  in 
turn  made  the  displays  less  intrusive  and  thus  more  acceptable  to  users. 

Tactile  displays  have  achieved  success  as  communication  systems  in  a  variety  of  contexts  in  which  it  has  been 
shown  that  vibrotactile  cues  delivered  to  the  skin  can  provide  spatial  information  about  the  environment  to  pilots,  or 
alert  the  operator  of  a  vehicle  of  an  impending  collision.  The  displays  are  typically  composed  of  a  matrix  of 
electromechanical  actuators  (known  as  tactors)  that  are  mounted  in  a  vest  or  waistband  and  are  sequentially 
activated  to  provide  information  about  a  person's  spatial  orientation  or  the  position  or  movement  of  a  vehicle.  In 
most  of  these  applications,  the  factors  are  activated  at  a  fixed  frequency  and  amplitude  and  the  number  and  location 
of  the  tactors  simultaneously  active  is  used  to  convey  information.  Tactile  displays  have  also  been  employed  in 
environments  in  which  the  visual  and  auditory  communication  channels  are  heavily  taxed  or  in  which  visual 
displays  are  inappropriate  because  the  operator  is  involved  in  other  activities  that  require  attention.  In  these  latter 
situations,  the  sense  of  touch  provides  a  communication  channel  that  is  direct  salient,  and  private. 

The  goal  of  the  research  conducted  for  the  Advanced  Decision  Architectures  Collaborative  Technology  Alliance 
(CTA)  was  to  design  and  build  wireless-controlled  wearable  tactile  displays  that  could  be  used  to  evaluate  the 
contexts  and  conditions  in  which  tactile  communication  facilitated  soldier  performance  in  the  battlefield,  in  vehicle 
or  robot  control,  and  in  information  exchange  during  planning  for  future  combat  missions.  The  domains  in  which  it 
was  envisaged  that  the  displays  would  be  used  were  those  that  required  that  the  hands  were  free  for  other  activities 
(e.g.  holding  a  weapon,  driving  a  vehicle).  The  display  was  therefore  designed  so  that  it  could  be  worn  on  the  arms, 
legs,  or  torso.  The  research  at  MIT  focused  on  five  issues:  Can  tactile  signals  be  used  to  provide  spatial  cues  about 
the  environment  that  are  accurately  localized?  How  does  the  location  and  configuration  of  the  tactile  display 
influence  the  ability  of  the  user  to  identify  tactile  patterns?  What  is  the  maximum  size  of  a  tactile  vocabulary  that 
could  be  used  for  communication?  Which  characteristics  of  vibrotactile  signals  are  optimal  for  generating  a  tactile 
vocabulary?  Can  a  set  of  Army  Hand  and  Arm  Signals  be  translated  into  tactile  signals  that  are  accurately  identified 
when  the  user  is  involved  in  concurrent  tasks?  Within  this  framework,  a  series  of  laboratory  and  field  studies  was 
conducted. 

The  tactile  displays  fabricated  at  MIT  were  made  available  to  other  partners  in  the  CTA  and  assistance  was  provided 
to  help  partners  conduct  experiments.  Studies  conducted  at  ARL  that  have  used  the  MIT  tactile  display  include  those 
that  have  done  the  following:  investigated  the  efficacy  oftactile  and  multimodal  alerts  on  decision  making  by  Army 
Platoon  Leaders;  analyzed  the  effectiveness  of  tactile  cues  in  target  search  and  localization  tasks  and  when 
controlling  robotic  swarms;  evaluated  Soldiers'  abilities  to  interpret  and  respond  to  tactile  cues  while  they  navigated 
an  Individual  Movement  Techniques  (IMT)  course;  and  measured  the  effects  of  tactile  cues  on  target  acquisition  and 
workload  of  Commanders  and  Gunners  and  determined  the  detectability  of  vibrotactile  cues  while  combat  assault 
maneuvers  were  being  performed.  The  MIT  tactile  displays  have  also  been  incorporated  into  multi-modal  platforms 
developed  by  the  University  of  Michigan,  ArtisTech  in  the  CTA  test  bed,  and  Alion  MA&D  for  a  robotics  control 
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environment.  Finally,  a  comprehensive  review  of  tactile  displays  was  written  in  collaboration  with  Nadine  Sarter  at 
the  University  of  Michigan. 


Jones,  L.  A.,  &  Ho,  H.-N.  (2008).  Warm  or  Cool,  Large  or  Small?  The  Challenge  of  Thermal  Displays.  IEEE 
Transactions  on  Haptics,  1,  53-70. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Evaluation/methodology,  haptic  i/o,  human  information  processing,  user  interfaces. 

Summary  abstract: 

Thermal  displays  have  been  developed  to  present  thermal  cues  to  the  hand  to  facilitate  object  recognition  in  virtual 
environments  or  in  teleoperated  robotic  systems.  This  review  focuses  on  this  application  domain  of  thermal  displays 
and  considers  the  models  developed  to  simulate  the  thermal  interaction  between  an  object  and  the  hand  as  they  make 
contact.  An  overview  of  thermal  perception  and  the  mechanisms  underlying  the  processing  of  thermal  information  is 
provided  to  give  a  framework  for  analyzing  the  design  of  thermal  displays.  The  models  developed  to  simulate 
thermal  feedback  are  examined  together  with  a  description  of  the  implementation  of  these  models  in  thermal 
displays.  The  domains  in  which  thermal  displays  have  been  used  are  described;  this  includes  the  simulation  of 
material  properties,  the  recreation  of  large-scale  thermal  effects  in  virtual  environments,  the  encoding  of  abstract 
concepts,  and  the  use  of  thermal  feedback  in  interactive  art.  The  review  concludes  by  considering  the  advantages 
and  challenges  associated  with  using  thermal  displays  in  these  diverse  areas. 


Jones,  L.  A.,  Kunkel,  J.,  &  Piateski,  E.  (2009).  Vibrotactile  Pattern  Recognition  on  the  Arm  and  Back.  Perception, 
38,  52-68. 
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Summary  abstract: 

A  series  of  experiments  was  conducted  to  evaluate  the  effectiveness  with  which  a  tactile  display  mounted  on  either 
the  forearm  or  the  back  can  be  used  to  communicate  simple  instructions  and  commands.  In  the  first  two  sets  of 
experiments,  participants  identified  a  vibrotactile  pattern  using  a  visual  template  that  represented  the  pattern  of 
activation.  For  the  patterns  displayed  on  the  forearm,  accuracy  depended  on  the  specific  set  of  patterns  presented  and 
ranged  from  30%  to  96%  correct  for  the  individual  patterns.  In  a  second  series  of  experiments,  seven  hand-and-arm 
signals  that  are  used  to  communicate  in  military  contexts  were  converted  into  tactile  representations  that  were 
displayed  on  the  back.  These  were  identified  accurately  (98%  correct)  and,  when  only  the  picture  of  the  hand  signal 
was  available,  participants  achieved  a  recognition  rate  of  75%  correct.  A  further  study  with  these  seven  patterns 
indicated  that  participants  were  still  able  to  identify  the  patterns  accurately  (92%  correct)  when  they  were  engaged  in 
a  concurrent  physical  or  cognitive  task.  The  results  indicate  the  importance  of  evaluating  tactile  communication  in 
the  context  of  the  specific  patterns  or  messages  that  will  be  conveyed,  and  that  with  the  judicious  selection  of  tactile 
patterns  both  the  arm  and  back  provide  a  functional  substrate  for  tactile  communication. 


Jones,  L.  A.,  Lockyer,  B.,  &  Piateski,  E.  (2006).  Tactile  Display  and  Vibrotactile  Recognition  on  the  Torso. 
Advanced  Robotics,  20(12),  1359-1374. 
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Summary  abstract: 

A  wirelessly  controlled  tactile  display  has  been  designed,  fabricated  and  tested  for  use  as  a  navigation  aid.  The 
display  comprises  a  4  x  4  array  of  vibrating  motors  that  is  mounted  on  a  waist  band  and  stimulates  the  skin  across 
the  lower  back.  Three  types  of  electromechanical  actuators  were  evaluated  for  use  in  the  display;  based  on  their 
mechanical  performance  and  power  requirements,  two  of  these  motors  were  then  used  to  fabricate  tactile  displays. 
The  performance  of  the  displays  and  the  wireless  tactile  control  units  was  assessed  experimentally  by  having 
subjects  identify  which  of  eight  possible  vibrotactile  patterns  was  presented  to  the  lower  back.  The  results  indicated 
that  subjects  could  recognize  the  vibrotactile  patterns  with  almost  perfect  accuracy  and  that  there  was  no  difference 
between  the  two  types  of  motor  used  for  the  displays.  Moreover,  the  ability  to  recognize  the  pattern  of  vibrotactile 
stimulation  was  superior  on  the  back  as  compared  to  the  forearm.  A  further  experiment  confirmed  that  the  tactile 
display  can  be  used  as  a  navigation  aid  outdoors  and  that  the  vibrotactile  patterns  presented  can  be  interpreted  as 
directional  or  instructional  cues  with  almost  perfect  accuracy 


Jones,  L.  A.,  &  Nakamura,  M.  (2003).  Haptic  Displays:  The  Tactile  Vest.  In  In  Proceedings  of  Collaborative 
Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  - 
May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

This  research  is  focused  on  the  development  of  a  torso  based  haptic  display  that  can  be  used  to  present  information 
about  orientation,  direction  of  movement,  or  threat  location  to  a  soldier  in  the  battlefield.  The  requirements  for  this 
display  are  that  it  can  be  worn  on  a  mobile  operator,  has  high  force  generating  capabilities,  is  light  weight  and 
robust.  A  tactile  vest  based  on  contractile  shape-memory  alloy  fibers  is  being  developed  to  meet  these  objectives, 
and  to  make  use  of  an  underutilized  sensory  channel,  while  leaving  the  hands  free  for  other  tasks.  Experiments 
conducted  with  nickel-titanium  fibers  in  a  prototype  display  in  which  a  pin  of  non-conducting  material  is  displaced 
as  the  fiber  contracts  indicate  that  the  forces  and  displacements  produced  by  these  fibers  greatly  exceed  the 
sensitivity  thresholds  of  the  human  torso.  An  array  of  these  fibers  and  pins  has  been  designed  and  is  now  being  built 
to  test  optimal  spacing  and  stimulation  requirements  for  human  use. 


Jones,  L.  A.,  &  Ray,  K.  (2009).  Localization  and  Pattern  Recognition  with  Tactile  Displays.  In  In  Proceedings  of 
IEEE  Proceedings  of  the  Symposium  on  Haptic  Interfaces  for  Virtual  Environment  and  Teleoperator  Systems,  33- 
39. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
CR  Categories:  G.3  [Mathematics  of  Computing]:  Probability  and  Statistics  -  Experimental  Design;  HI. 2  [Model 
and  Principles]:  User/Machine  System;  H.5.2  [Information  Interfaces  and  Presentation]:  User  Interfaces  -  Theory 
and  methods;  Usercentered  design 

Keywords:  tactile  display,  touch,  torso,  localization,  tactons 
Summary  abstract: 

A  set  of  four  experiments  was  conducted  to  evaluate  tactile  localization  and  tactile  pattern  recognition  on  the  torso. 

A  onedimensional  eight-tactor  display  and  a  two-dimensional  16-tactor  display  were  used  to  present  tactile  cues  to 
the  waist  and  back  respectively.  The  results  from  the  spatial  localization  experiments  indicated  that  a  display  with 
eight  tactors  mounted  circumferentially  around  the  waist  can  provide  tactile  cues  that  are  perceived  very  accurately 
in  terms  of  the  location  of  stimulation.  In  contrast,  the  16-tactor  array  on  the  back  was  found  to  be  inadequate  to 
support  precise  spatial  mapping,  but  an  array  with  fewer  elements  could  provide  such  spatial  cues.  The  second  set  of 
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experiments  evaluated  tactile  pattern  recognition  around  the  waist  and  on  the  back  with  the  objective  of  determining 
what  types  of  tactile  stimuli  could  be  useful  for  creating  tactons.  The  tactile  display  used  in  these  experiments 
stimulated  the  skin  at  a  fixed  frequency  and  amplitude  and  varied  the  number  and  location  of  tactors  simultaneously 
active  to  convey  information.  For  both  the  waist  and  back,  tactile  patterns  were  identified  with  high  accuracy,  99% 
and  95%  correct  response  rate  respectively.  These  findings  suggest  that  simple  navigational  and  instructional 
commands  can  be  presented  tactually  on  the  torso. 


Jones,  R.  E.  T.,  Connors,  E.  S.,  &  Endsley,  M.  R.  (2009).  Incorporating  the  Human  Analyst  into  the  Data  Fusion 
Process  by  Modeling  Situation  Awareness  Using  Fuzzy  Cognitive  Maps.  In  In  Proceedings  of  12th  International 
Conference  on  Information  Fusion.  Seattle,  WA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Situation  awareness,  fizzy  cognitive  maps,  high  level  fusion,  and  human  cognitive  processes. 

Summary  abstract: 

Current  data  fusion  models  lack  the  capability  of  fully  supporting  the  cognitive  processes  of  the  human  analyst.  The 
data  fusion  community  has  expressed  a  need  to  better  incorporate  users  within  their  models.  The  purpose  of  this 
paper  is  to  describe  how  to  use  fizzy  logic  to  develop  a  data  fusion  model  that  supports  situation  awareness  (SA). 
Developing  this  model  based  on  the  formal  representation  of  the  analyst  provided  by  the  Goal-  Directed  Task 
Analysis  (GDTA)  methodology  advances  current  data  fision  models  because  it  provides  valuable  insight  on  how  to 
effectively  support  human  cognition  within  the  data  fusion  process. 


Jones,  R.  E.  T.,  Connors,  E.  S.,  Mossey,  M.  E.,  Hyatt,  J.  R.,  Hansen,  N.  J.,  &  Endsley,  M.  R.  (2009).  Modeling 
Situation  Awareness  for  Army  Infantry  Platoon  Leaders  Using  Fuzzy  Cognitive  Mapping  Techniques.  In  In 
Proceedings  of  19th  Behavior  Representation  in  Modeling  and  Simulation  (BRIMS). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Situation  Awareness,  Fuzzy  Cognitive  Maps,  Army  Infantry  Platoon,  METT-TC,  Modeling  and  Simulation 

Summary  abstract: 

This  paper  describes  work  on  the  development  of  an  actionable  model  of  situation  awareness  for  Army  infantry 
platoon  leaders  using  fizzy  cognitive  mapping  techniques.  Developing  this  model  based  on  the  formal 
representation  of  the  platoon  leader  provided  by  the  Goal-Directed  Task  Analysis  (GDTA)  methodology  advances 
current  cognitive  models  because  it  provides  valuable  insight  on  how  to  effectively  support  human  cognition  within 
the  decision-making  process.  We  describe  the  modeling  design  approach  and  discuss  validating  the  model  using  the 
VBS2  simulation  environment. 


Josephson,  J.  R.  (2008).  Abductive  Inferencing  for  Integrating  Information  from  Human  and  Robotic  Sources.  In  In 
Proceedings  of  Fusion  2008  Conference,  Cologne,  Germany. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
abductive  inference,  hard  source,  soft  source, 
level- 1  fusion,  level-3  fision,  credibility,  veracity. 

Summary  abstract: 

Abductive  inference  (best-explanation  reasoning)  is  a  useful  conceptual  framework  for  analyzing  and  implementing 
the  inferencing  needed  to  integrate  information  from  human  and  robotic  sources.  Inferencing  proceeds  from  reports, 
to  explanations  for  these  reports,  given  in  terms  of  hypothesized  real-world  entities  and  the  processes  by  which  the 
entities  lead  to  the  reports. 
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Reports  from  humans  and  robotic  sources  are  subject  to  different  kinds  of  corruption,  so  they  require  different 
treatment  as  sources  of  evidence.  The  best  explanation  for  a  certain  report  might  be  that  it  presents  a  reliable 
statement  that  results  from  a  chain  of  causality  from  the  events  reported,  to  their  effects  on  human  or  robotic  senses, 
and  from  there  through  transduction,  processing,  and  reporting.  Confidence  in  this  explanation  will  be  undercut  by 
evidence  supporting  a  rival  explanation,  such  as  one  involving  error  or  intended  deception. 


Josephson,  J.  R.,  Chandrasekaran,  B.,  &  Carroll,  M.  (2003).  Toward  a  Generic  Architecture  for  Multisource 
Information  Fusion.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and 
Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University 
College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

This  paper  sketches  a  design  for  software  for  building,  composing,  and  maintaining  information  fusion  application 
software.  It  is  intended  to  support  a  broad  range  of  information  fusion  applications  that  vary  with  respect  to  types  of 
information  sources  (imaging  sensors,  human  reports,  etc.)  and  with  respect  to  operational  demands  (e.g.,  timeliness, 
reliability,  availability  of  human  attention).  The  design  relies  on  abductive  inference,  compositional  modeling, 
model-based  prediction,  anomaly-driven  processing,  and  need-driven  processing.  Some  main  components  of  the 
design  have  previously  been  demonstrated  in  prototype  systems  that  have  shown  good  performance. 


Josephson,  J.  R.,  Eckroth,  J.,  &  Miller,  T.  N.  (2009).  Estimation  of  Adversarial  Social  Networks  by  Fusion  of 
Information  from  a  Wide  Range  of  Sources.  In  In  Proceedings  of  12th  International  Conference  on  Information 
Fusion,  Fusion  2009.  Seattle,  WA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Information  fusion,  soft  information,  social  networks,  biometrics,  asymmetric  warfare,  urban  operations 

Summary  abstract: 

A  data  structure  is  described  that  serves  to  define  a  target  structure  for  estimating  social  networks.  It  represents  who 
knows  whom,  the  strength  and  polarity  of  associations,  and  levels  of  confidence  that  linked  individuals  are  actually 
personally  acquainted.  This  network,  which  represents  social  structure,  is  embedded  in  a  more  inclusive  network 
structure  that  also  represents  vehicles,  places  and  organizations.  This  broader  structure  can  be  used  to  accumulate 
information  to  enable  automated  inferencing  to  assist  human  processing  in  estimating  the  social  network  of  interest. 
How  portions  of  this  inferencing  can  be  done  is  briefly  described,  and  a  scenario  is  given  that  illustrates  the  use  of 
this  kind  of  fused  information  to  make  a  decision.  We  also  review  a  range  of  hard  and  soft  information  sources  with 
emerging  value  for  estimating  adversarial  social  networks,  and  describe  how  these  sources  can  be  used  for  the 
purpose. 


Josephson,  J.  R.,  Sobieski,  A.,  &  Chandrasekaran,  B.  (2003).  Detection  of  Social  Order:  Finding  Dense  and  Isolated 
Groups  Using  Message  Traffic.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science 
and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University 
College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

This  paper  reports  on  the  discovery  of  what  appears  so  far  to  be  a  very  effective  computational  strategy  for  finding 
suspects  belonging  to  relatively  dense  and/or  isolated  groups.  In  experiments  so  far,  this  “S2B2-S1B1”  filtering 
strategy  has  been  both  sensitive  (does  not  miss  groups)  and  selective  (does  not  give  many  false  positives).  We 
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describe  analysis  and  experiments  to  understand  why  this  strategy  works  as  well  as  it  does,  and  to  discover  its  limits. 
This  strategy  might  be  integrated  into  multisource  systems  for  finding  and  analyzing  enemy  command  and  control 
systems,  and  finding  their  points  of  vulnerability. 


Juarez,  O.,  &  Gonzalez,  C.  (2003).  Masa:  Meta-Architecture  for  Situation  Awareness.  In  In  Proceedings  of  Behavior 
Representation  in  Modeling  and  Simulation  Conference.  Scottsdale,  AZ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

MASA: 

•  A  Meta- Architecture  to  model  SA  in  different  domains 

•  Helps  modularize  the  knowledge  and  the  system  functionality  to  represent  SA 

•  First  initial  development:  Models  of  Situation  Awareness  in  complex  battle  situations. 


Juarez-Espinosa,  O.,  &  Gonzalez,  C.  (2003).  Situation  Awareness  of  Commanders:  A  Cognitive  Model.  In  In 
Proceedings  of  Conference  on  Behavior  Representation  in  Modeling  and  Simulation.  Arlington,  VA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Situation  Awareness,  Cognitive  Model 

Summary  abstract: 

This  paper  discusses  a  computational  model  of  situation  awareness  (SA)  for  military  command  and  control  in 
complex  battle  situations.  Cognitive  modeling  is  a  research  method  that  does  not  forgive  vagueness.  How  do 
military  commanders  create  awareness  in  a  highly  complex  and  uncertain  world?  The  model  described  in  this  paper 
simulates  computationally  some  of  the  cognitive  operations  performed  by  commanders  during  the  evaluation  of  a 
complex  battle  situation. 

Although  the  set  of  cognitive  processes  that  support  SA  are  still  not  well  understood,  we  have  hypothesized  a 
metaarchitecture  involving:  information  gathering,  assessment,  and  alternative  generation.  The  cognitive  model 
reported  here  paves  the  road  towards  a  more  complete  and  valid  representation.  The  model  was  implemented  using 
ACT-R,  a  cognitive  architecture,  and  tested  with  scenarios  running  in  OTB,  a  simulation  tool  for  war  scenarios. 


Juarez-Espinosa,  O.,  &  Gonzlaez,  C.  (2003).  Using  Petri  Nets  to  Analyze  Cognitive  Multitasking  Domains.  In  In 
Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the 
Objective  Force.  April  29  -  May  1,  2003  (pp.  175-177).  College  Park,  MD:  U.S.  Army  Research  Laboratory. 

Key  Words: 

Human  Performance 

Summary  abstract: 

This  paper  describes  the  use  of  Petri  nets  (PN)  to  perform  analysis  of  concurrency,  synchronization,  and  conflict  in 
multitasking  domains.  Those  analyses  precede  the  design  and  implementation  of  cognitive  models.  Petri  nets  (PN) 
are  useful  tools  to  model  multitasking  and  attention  sharing.  To  demonstrate  the  usefulness  of  this  tool  in  the  process 
of  designing  a  cognitive  mode,  we  present  an  example  that  involves  four  tasks.  A  prototype  to  analyze  two  tasks  that 
share  the  visual  resources  was  implemented  using  a  PN  tool.  The  manipulation  of  parameters  for  delay  and  priorities 
in  the  transitions  provided  us  with  good  information  about  this  task  which  enabled  us  to  design  and  implement  the 
cognitive  model. 


Jung,  H.,  Bradshaw,  J.  M.,  Kulkami,  S.,  Breedy,  M.,  Bunch,  L.,  Paul  Feltovich,  Jeffers,  R.,  Johnson,  M.,  Lott,  J., 
Suri,  N.,  Taysom,  W.,  Tonti,  G.,  &  Uszok,  A.  (2004).  An  Ontology-Based  Representation  for  Policy-Governed 
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Adjustable  Autonomy.  In  In  Proceedings  of  American  Association  for  Artificial  Intelligence  (AAAI)  Spring 
Symposium.  Stanford,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Policies  are  a  means  to  dynamically  regulate  the  behavior  of  system  components  without  changing  code  nor 
requiring  the  cooperation  of  the  components  being  governed.  By  changing  policies,  a  system  can  be  continuously 
adjusted  to  accommodate  variations  in  externally  imposed  constraints  and  environmental  conditions.  KAoS  policy 
and  domain  services  rely  on  an  OWL  ontology  of  the  computational  environment,  application  context,  and  the 
policies  themselves  that  enables  runtime  extensibility  and  adaptability  of  the  system,  as  well  as  the  ability  to  analyze 
policies  relating  to  entities  described  at  different  levels  of  abstraction.  Besides  the  currently  implemented  conflict 
detection  and  resolution  methods,  we  are  developing  an  approach  to  determine  how  and  when  to  make  policy 
changes  based  on  adjustable  autonomy  considerations.  This  approach  relies  heavily  on  the  information  contained  in 
the  KAoS  Policy  Ontologies. 


Kaber,  D.  B.,  &  Endsley,  M.  R.  (2004).  The  Effects  of  Level  of  Automation  and  Adaptive  Automation  on  Human 
Performance,  Situation  Awareness  and  Workload  in  a  Dynamic  Control  Task.  Theoretical  Issues  in  Ergonomic 
Science,  5(2),  113-153. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Level  of  automation  (LOA);  adaptive  automation  (AA);  human-centred  automation; 
situation  awareness;  mental  workload;  embedded-secondary  task. 

Summary  abstract: 

This  paper  extends  previous  research  on  two  approaches  to  human-centred  automation:  (1)  intermediate  levels  of 
automation  (LOAs)  for  maintaining  operator  involvement  in  complex  systems  control  and  facilitating  situation 
awareness;  and  (2)  adaptive  automation  (AA)  for  managing  operator  workload  through  dynamic  control  allocations 
between  the  human  and  machine  over  time.  Some  empirical  research  has  been  conducted  to  examine  LOA  and  AA 
independently,  with  the  objective  of  detailing  a  theory  of  human-centred  automation.  Unfortunately,  no  previous 
work  has  studied  the  interaction  of  these  two  approaches,  nor  has  any  research  attempted  to  systematically  determine 
which  LOAs  should  be  used  in  adaptive  systems  and  how  certain  types  of  dynamic  function  allocations  should  be 
scheduled  over  time.  The  present  research  briefly  reviews  the  theory  of  humancentred  automation  and  LOA  and  AA 
approaches.  Building  on  this  background,  an  initial  study  was  presented  that  attempts  to  address  the  conjuncture  of 
these  two  approaches  to  human-centred  automation.  An  experiment  was  conducted  in  which  a  dual-task  scenario 
was  used  to  assess  the  performance,  SA  and  workload  effects  of  low,  intermediate  and  high  LOAs,  which  were 
dynamically  allocated  (as  part  of  an  AA  strategy)  during  manual  system  control  for  various  cycle  times  comprising 
20,  40  and  60%  of  task  time.  The  LOA  and  automation  allocation  cycle  time  (AACT)  combinations  were  compared 
to  completely  manual  control  and  fully  automated  control  of  a  dynamic  control  task  performed  in  conjunction  with 
an  embedded  secondary  monitoring  task.  Results  revealed  LOA  to  be  the  driving  factor  in  determining  primary  task 
performance  and  SA.  Low-level  automation  produced  superior  performance  and  intermediate  LOAs  facilitated 
higher  SA,  but  this  was  not  associated  with  improved  performance  or  reduced  workload.  The  AACT  was  the  driving 
factor  in  perceptions  of  primary  task  workload  and  secondary  task  performance.  When  a  greater  percentage  of 
primary  task  time  was  automated,  operator  perceptual  resources  were  freed-up  and  monitoring  performance  on  the 
secondary  task  improved.  Longer  automation  cycle  times  than  have  previously  been  studied  may  have  benefits  for 
overall  human-machine  system  performance.  The  combined  effect  of  LOA  and  AA  on  all  measures  did  not  appear 
to  be  'additive'  in  nature.  That  is,  the  LOA  producing  the  best  performance  (low  level  automation)  did  not  do  so  at 
the  AACT,  which  produced  superior  performance  (maximum  cycle  time).  In  general,  the  results  are  supportive  of 
intermediate  LOAs  and  AA  as  approaches  to  human-centred  automation,  but  each  appears  to  provide  different 
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benefits  to  human-machine  system  performance.  This  work  provides  additional  information  for  a  developing  theory 
of  human-centred  automation. 


Kamali,  K.,  Fan,  X.,  &  Yen,  J.  (2006).  Multiparty  Proactive  Communication:  A  Perspective  for  Evolving  Shared 
Mental  Models.  In  In  Proceedings  of  the  Twenty  First  National  Conference  on  Artificial  Intelligence  (AAAI-06), 
Boston,  MA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Helping  behavior  in  effective  teams  is  enabled  by  some  overlapping  “shared  mental  models”  that  are  developed  and 
maintained  by  members  of  the  team.  In  this  paper,  we  take  the  perspective  that  multiparty  “proactive” 
communication  is  critical  for  establishing  and  maintaining  such  a  shared  mental  model  among  teammates,  which  is 
the  basis  for  agents  to  offer  proactive  help  and  to  achieve  coherent  teamwork.  We  first  provide  formal  semantics  for 
multiparty  proactive  performatives  within  a  team  setting.  We  then  examine  how  such  performatives  result  in  updates 
to  mental  model  of  teammates,  and  how  such  updates  can  trigger  helpful  behaviors  from  other  teammates. 


Kaste,  R.,  Heilman,  E.,  &  Hoffman,  R.  (2007).  Concept  Map  Value  Propagation  for  Tactical  Intelligence.  In  In 
Proceedings  of  Twelfth  International  Command  and  Control  Research  and  Technology  Symposium  (12th  ICCRTS), 
19-21 

June  2007,  Newport,  RI.  DTIC  ADA481481. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Fields  and  Groups: 

150400  -  Military  Intelligence 

230200  -  Human  Factors  Engineering  &  Man  Machine  System 
Descriptors: 

(U)  *MAN  MACHINE  SYSTEMS,  *TACTICAL  INTELLIGENCE,  PROPAGATION,  SYMPOSIA, 

NODES,  SOFTWARE  TOOLS,  KNOWLEDGE  MANAGEMENT,  SITUATIONAL  AWARENESS, 

DATA  LINKS,  COMMAND  AND  CONTROL  SYSTEMS,  MAPS 
Identifiers: 

(U)  *CMAPS(CONCEPT  MAPS),  INFERENCE  NETWORKS 
Summary  abstract: 

Command  and  control  analysts  increasingly  apply  traditionally  unassociated  concepts  to  situation  understanding. 
Techniques  are  needed  to  tailor  knowledge  acquisition  resource  allocation  according  to  probable  value  of 
information,  both  inferring  an  answer  to  a  question  using  knowledge  at  hand  and  quickening  intelligence  efforts  to 
fill  in  gaps.  Concept  Maps  ("Cmaps")  are  a  form  of  meaningful  diagram  that  represents  concepts  as  nodes,  linked  by 
specified  relationships.  This  paper  discusses  research  into  methods  for  negotiating  and  updating  Cmaps  by 
accounting  for  both  nodal  data  and  links.  Cmaps  provide  the  flexibility  to  represent  at  a  meaningful  level 
contemporary  tactical  knowledge  not  lending  itself  to  conventional  data  structures.  In  some  senses  Cmapping 
generalizes  the  notion  of  an  inference  network,  a  set  of  propositions  organized  with  rules  directing  information 
propagation  and  combining  antecedents  to  update  consequents.  We  are  attempting  to  develop  a  mathematical  system 
for  organized  navigation  of  a  Cmap,  driven  by  expected  variability  in  the  value  of  a  datum  and  cost  to  get  a  new 
value.  We  use  the  CmapTools  software  developed  with  DoD  support  at  the  Institute  of  Human  and  Machine 
Cognition  as  a  structural  basis  for  creating  and  assessing  tactical  Cmaps.  The  paper  sets  forth  Cmap  construction, 
analytical  philosophy,  and  methodology  development. 


Kaste,  R.,  O'May,  J.,  Heilman,  E.,  Josephson,  J.  R.,  &  Chandrasekaran,  B.  (2003).  From  Simulation  to  Insights: 
Experiments  in  the  Use  of  a  Multi -Criterial  Viewer  to  Develop  Understanding  of  the  Coa  Space.  In  In  Proceedings 
of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective 
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Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research 
Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

Building  on  advances  in  modeling  and  simulation,  the  U.S.  Army  Research  Laboratory  (ARL)  has  developed  a 
capability  for  simulating  detailed  Courses  of  Action  (CO As).  The  Ohio  State  University  (OSU)  has  developed  a 
multi-criterial  decision  technology  known  as  the  Seeker-Filter- Viewer.  In  this  paper,  we  report  on  initial  results  of  a 
collaborative  effort  between  researchers  at  ARL  and  OSU  in  experimenting  with  the  decision  tool  for  mining  ARL 
combat  simulation  data  to  gain  battle-planning  insights.  The  capability  of  simulating  detailed  COAs  opens  up 
possibilities  of  mining  collected  data  for  insights.  Decision  support  systems  could  assist  commanders  in  examining 
simulation  data  for  relationships  between  COA  structure  and  various  battle  objectives.  The  synergy  of  data  mining 
tools,  high  performance  computing,  and  advanced  high-resolution  combat  simulation  has  potential  to  lead  battle 
planners  to  new  insights  for  imminent  combat,  translating  to  improved  battlefield  assessments  and  expedient 
modification  of  COAs. 


Kiekel,  P.  A.,  Cooke,  N.  J.,  Foltz,  P.  W.,  Gorman,  J.  C.,  &  Martin,  M.  J.  (2002).  Some  Promising  Results  of 
Communication-Based  Automatic  Measures  of  Team  Cognition.  In  In  Proceedings  of  46th  Annual  Meeting  of  the 
Human  Factors  and  Ergonomics  Society.  Baltimore,  MD.  (pp.  298-302). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Some  have  argued  that  the  most  appropriate  measure  of  team  cognition  is  a  holistic  measure  directed  at  the  entire 
team.  In  particular,  communication  data  are  useful  for  measuring  team  cognition  because  of  the  holistic  nature  of  the 
data,  and  because  of  the  connection  between  communication  and  declarative  cognition.  In  order  to  circumvent  the 
logistic  difficulties  of  communication  data,  the  present  paper  proposes  several  relatively  automatic  methods  of 
analysis.  Four  data  types  are  identified,  with  low-level  physical  data  vs.  content  data  being  one  dimension,  and 
sequential  vs.  static  data  being  the  other.  Methods  addressing  all  four  of  these  data  types  are  proposed,  with  the 
exception  of  static  physical  data.  Latent  Semantic  Analysis  is  an  automatic  method  used  to  assess  content,  either 
statically  or  sequentially.  PRONET  is  useful  to  address  either  physical  or  content-based  sequential  data,  and  we 
propose  CHUMS  to  address  sequential  physical  data.  The  usefulness  of  each  method  to  predict  team  performance 
data  is  assessed. 


Kiekel,  P.  A.,  Cooke,  N.  J.,  Foltz,  P.  W.,  &  Shope,  S.  M.  (2001).  Automating  Measurement  of  Team  Cognition 
through  Analysis  of  Communication  Data.  In  M.J.  Smith,  G.  Salvendy,  D.  Harris,  &  R.J.  Koubek  (Eds.),  Usability 
Evaluation  and  Interface  Design.  Pp.  1382  -  1386.  Mahwah,  NJ:  Lawrence  Erlbaum  Associates,  Inc. 

Key  Words: 
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Summary  abstract: 

In  this  paper  we  propose  a  general  methodological  approach  for  semi-automntically  assessing  team  cognition  using 
communicntion  data.  The  approach  rests  on  four  premises:  I)  analyzing  communication  data  is  a  means  of  assessing 
team  cognition.  2)  both  substantive  content  and  physical  quantity  measures  of  communication  are  needed.  3) 
sequential  flow  methods  are  especially  helpful  to  effectively  make  use  of  communication  data,  and  4)  analysis  of 
both  data  types  can  be  automated  with  contemporary  tools  ,  both  statically  and  sequentially.  We  begin  by  illust 
rating  the  first  three  points.  Next,  we  briefly  review  commonly  employed  methods  of  communication  analysis,  and 
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note  the  difficulties  of  analyzing  such  data.  We  then  suggest  that  appropriate  automatic  methods  for  analyzing 
communication  data  are  becoming  increasingly  available,  and  we  give  some  examples  of  our  approach.  Finally,  we 
conclude  by  discussing  the  implications  of  Ihis  approach,  especially  for  team  training  and  groupware  design. 


Klein,  G.  (2003).  Chapter  12.  Executive  Intent:  How  to  Communicate  Your  Intuitions.  In  G.  Klein  (Ed.),  Intuition 
at  Work.  New  York:  Doubleday. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

One  of  the  difficulties  in  using  intuitive  decision  making  is  that  you  may  struggle  to  express  exactly  what  your 
intuition  is  telling  you.  It  isn't  enough  to  make  great  decisions  if  you  can't  get  them  implemented.  We  run  into  this 
problem  on  both  ends:  when  we  try  to  communicate  our  intent  to  our  subordinates  and  when  we  struggle  to  interpret 
the  intent  expressed  by  our  bosses. 

We  need  to  find  ways  to  communicate  our  intentions  clearly  to  our  subordinates.  When  they  don't  understand  the 
reasons  behind  our  instructions,  they're  ill-equipped  to  respond  to  unexpected  problems  or  questions.  That's  what 
made  it  hard  to  figure  out  how  to  respond  to  George  in  the  second  phase  of  the  decision  game. 

And  when  we  are  receiving  directions  and  intentions  from  someone  else,  such  as  Jennifer  in  the  decision  game 
above,  we  have  to  reach  beyond  the  words  to  determine  what  the  person  wants. 


Klein,  G.,  &  Armstrong,  A.  A.  (2005).  Chapter  35.  Critical  Decision  Method.  In  N.  Stanton,  A.  Hedge,  K. 
Brookhuis,  E.  Salas,  &  H.  Hendrick  (Eds.),  Handbook  of  Human  Factors  and  Ergonomics  Methods  (pp.  1-8). 
London:  CRC  Press. 
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Critical  Decision  Method 
Background  and  Scenario 

•  Observation  of  Scenario 

•  Observations  of  Human-System 
Interactions 

•  Postobservation  Interviews 

Adapting  Critical  Decision  Method  to  Address  Critical 
Needs 

•  Within-Method  Adaptations 

•  Cross-Method 
Adaptations 

Standards,  Regulations,  and  Adaptation 
Identifying  an  Incident 

•  Creating  a  Detailed  Timeline 

•  Deepening  on  Decision  Points 

•  Probing  with  “What-If  ” 

Summary  abstract: 

Background  and  Application 

The  naturalistic  study  of  expert  decision  making  requires  an  in-depth  analysis  of  complex  cognitive  tasks.  It  involves 
the  exploration  of  the  use  of  perceptual  cues,  the  development  of  expert  knowledge,  and  the  evolution  of  expert 
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strategies.  This  type  of  exploration  is  most  effective  when  experts  are  studied  in  the  context  of  their  complex 
environments.  Cognitive  Task  Analysis  (CTA)  methods  for  eliciting  and  analyzing  the  aspects  of  expertise  needed 
to  perform  complex  tasks  have  been  evolving  over  the  past  two  decades. 

CTA  methods  must  be  highly  adaptable  for  two  reasons.  First,  the  data  collection  conditions  can  vary  from  one 
project  to  the  next,  and  therefore  the  methods  have  to  be  flexible.  Second,  CTA  methods  are  often  used  to  make  new 
discoveries,  which  mean  that  the  phenomena  of  interest  are  not  well  known  at  the  outset  of  the  research. 
Consequently,  the  CTA  methods  being  used  must  be  adjustable  to  take  advantage  of  what  has  been  learned  so  that 
they  can  better  focus  on  the  critical  issues. 

The  method  was  developed  more  than  15  years  ago  (Klein  et  al.,  1986)  and  continues  to  be  useful  for  conducting 
naturalistic  decision-making  research.  We  have  codified  the  procedures  for  conducting  Critical  Decision  Method 
(CDM)  projects,  but  at  the  same  time  we,  and  others  in  our  field,  continue  to  learn  how  to  adapt  the  CDM  by 
changing  the  way  it  is  executed  and  by  synthesizing  it  with  complementary  data-collection  tools  and  approaches. 
Procedure:  Overview  of  the  Critical  Decision  Method 

One  approach  to  cognitive  task  analysis  is  the  CDM  (Klein  et  ah,  1989;  Hoffman  et  ah,  1998).  The  CDM  was 
developed  as  an  extension  of  the  critical  incident  technique  (CIT)  (Flanagan,  1954)  and  uses  indepth  interviews  to 
gather  retrospective  accounts  of  challenging  incidents.  The  CDM  is  a  semistructured  interviewing  technique  for 
investigating  phenomena  that  rely  on  subtle  cues,  knowledge,  goals,  expectancies,  and  expert  strategies. 

The  CDM  does  not  use  a  strict  protocol  of  interview  questions.  It  is  structured  by  a  set  of  interview  phases  or 
“sweeps”  that  examine  the  incident  in  successively  greater  detail.  A  typical  CDM  session  requires  approximately  2 
hours  to  move  through  each  of  the  four  interview  sweeps: 

1.  Identify  a  complex  incident  that  has  the  potential  to  elicit  discoveries  about  cognitive  phenomena. 

2.  Create  a  detailed  incident  timeline  that  shows  the  sequence  of  events. 

3.  Deepen  strategies  for  managing  the  decision  points  embedded  in  the  timeline. 

4.  Probe  with  what-if  queries  to  elicit  potential  expert/novice  differences. 


Klein,  G.,  Feltovich,  P.  J.,  Bradshaw,  J.  M.,  &  Woods,  D.  D.  (2004).  Common  Ground  and  Coordination  in  Joint 
Activity.  In  W.B.  Rouse  and  K.R.  Boff  (Eds.),  Organizational  Simulation.  New  York:  John  Wiley  &  Sons,  Inc. 
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Summary  abstract: 

Generalizing  the  concepts  of  joint  activity  developed  by  Clark  (1996),  we  describe  key  aspects  of  team  coordination. 
Joint  activity  depends  on  interpredictability  of  the  participants’  attitudes  and  actions.  Such  interpredictability  is 
based  on  common  ground — pertinent  knowledge,  beliefs  and  assumptions  that  are  shared  among  the  involved 
parties.  Joint  activity  assumes  a  basic  compact,  which  is  an  agreement  (often  tacit)  to  facilitate  coordination  and 
prevent  its  breakdown.  One  aspect  of  the  Basic  Compact  is  the  commitment  to  some  degree  of  aligning  multiple 
goals.  A  second  aspect  is  that  all  parties  are  expected  to  bear  their  portion  of  the  responsibility  to  establish  and 
sustain  common  ground  and  to  repair  it  as  needed. 

We  apply  our  understanding  of  these  features  of  joint  activity  to  account  for  issues  in  the  design  of  automation. 
Research  in  software  and  robotic  agents  seeks  to  understand  and  satisfy  requirements  for  the  basic  aspects  of  joint 
activity.  Given  the  widespread  demand  for  increasing  the  effectiveness  of  team  play  for  complex  systems  that  work 
closely  and  collaboratively  with  people,  observed  shortfalls  in  these  current  research  efforts  are  ripe  for  further 
exploration  and  study. 


Klein,  G.,  &  Hoffman,  R.  R.  (2008).  Chapter  4.  Macrocognition,  Mental  Models,  and  Cognitive  Task  Analysis 
Methodology.  In  J.M.  Schraagen,  L.G.  Militello,  T.  Ormerod,  &  R.  Lipshitz  (Eds.),  Naturalistic  Decision  Making 
and  Macrocognition  (pp.  57-80).  Aldershot,  England:  Ashgate  Publishing,  Inc. 

Key  Words: 
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Summary  abstract: 

In  one  view  of  macro  cognition  (Klein  et  ai.,  2003),  the  "primary  functions"  are  things  that  domain  practitioners  say 
they  need  to  accomplish:  replanning,  sensemaking,  decision  making,  and  so  on.  The  "supporting  processes"  are 
cognitive  capacities  that  make  possible  the  achievement  of  the  primaries.  Mental  modeling  is  such  a  supporting 
process,  especially  critical  for  sensemaking  (Klein  et  ai.,  2006).  This  chapter  is  an  exploration  of  some  methods  for 
gathering  data,  representing,  and  studying  mental  models.  (In  this  chapter,  we  do  not  review  methods  of  knowledge 
elicitation,  broadly  conceived.  Reviews  have  been  provided  by  Cooke  (1992,  1994),  Crandall,  Klein,  and  Hoffman 
(2006),  Hoffman  and  Lintem  (2006),  Hoffman  et  al.  (1995),  and  Olson  and  Reuter  (1987).) 

Our  purpose  in  this  chapter  is  fairly  immediate  and  practical:  To  discuss  methods  and  methods  for  studying  mental 
models.  We  discuss  tried-and-true  methods,  but  we  also  present  some  ideas  about  new  methods,  with  an  invitation 
for  researchers  to  apply  and  evaluate  them.  This  is  a  practitioner's  account,  that  is,  a  description  of  a  battery  of 
techniques  that  researchers  have  developed  for  dealing  with  the  problems  associated  with  eliciting  and  refining 
meaningful  and  usable  accounts  of  expert  cognition.  Our  stance  is  "practitioner  confessional"  rather  than  a  dogmatic 
"this  is  how  it  should  be."  This  chapter  is  basically  an  invitation  to  naturalistic  empirical  inquiry:  If  one  were 
conducting  cognitive  task  analysis  (eTA)  on  experienced  domain  practitioners,  for  purposes  such  as  cognitive 
systems  engineering,  what  sorts  of  methods  might  be  used  to  reveal  mental  models? 

First,  however,  we  do  need  to  deal  with  some  philosophical  issues.  Any  treatment  of  mental  models  cOLdd  go  into 
great  length  about  issues  of  scientific  meaning,  validity,  relation  to  phenomenology,  and  so  on,  and  could  rehash  a 
considerable  literature  on  the  scientific  status  of  mental  models.  With  this  in  mind,  we  take  pains  to  be  succinct. 


Klein,  G.,  Moon,  B.,  &  Hoffman,  R.  R.  (2006).  Making  Sense  of  Sensemaking  1:  Alternative  Perspectives.  IEEE 
Intelligent  Systems,  22-25. 
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Summary  abstract: 

A  man  was  worried  about  his  72-year-old  father,  who  had  just  had  a  pacemaker  implanted.  The  man  believed  that 
his  father’s  condition  was  serious,  despite  reassurances  from  the  hospital  staff.  The  man’s  father  had  shortness  of 
breath,  cardiac  arrhythmia,  mild  congestive  heart  failure,  an  enlarged  heart,  water  retention,  mild  high  blood 
pressure,  mild  emphysema,  and  a  heart  valve  replacement  10  years  earlier.  The  combination  of  all  these  symptoms 
and  problems  seemed  ominous.  The  man  coaxed  a  physician  to  explain  what  was  going  on. 

The  physician  said  that  the  heart  valve  replacement  was  irrelevant.  Basically,  the  father  had  a  slightly  enlarged  heart. 
That  wasn’t  a  big  problem  except  that  the  area  of  enlargement  had  stretched  some  of  the  nerves  that  controlled  heart 
rate;  this  caused  the  cardiac  arrhythmia.  The  arrhythmia,  in  turn,  meant  that  the  father’s  heart  was  less  efficient  at 
maintaining  fluid  levels,  which  is  often  a  problem  of  aging.  So,  the  fluid  buildup  resulted  in  mild  congestive  heart 
failure  and  shortness  of  breath.  The  mild  emphysema  didn’t  help.  And  that’s  why  they  installed  the  pacemaker.  With 
that  simple  story,  the  various  data  elements  fit  together  in  a  coherent  causal  scheme,  satisfying  the  man  that  this  was 
a  treatable  problem  rather  than  a  cascading  breakdown  of  health. 

This  stoiy  is  one  of  many  that  researchers  use  to  illustrate  the  phenomenon  of  sensemaking.  Although  we  can  trace 
this  notion  to  the  early  1980s, 1  it  has  emerged  since  the  1990s  as  a  subject  for  organizational  research, 2^4 
educational  research,5  and  symposia  on  decision  making. 6  Sensemaking  has  become  an  umbrella  term  for  efforts  at 
building  intelligent  systems — for  example,  the  research  on  data  fusion  and  adaptive  interfaces.  7, 8  Research  requests 
are  frequently  issued  for  intelligent  systems  that  will 

•  automatically  fuse  massive  data  into  succinct  meanings, 

•  process  meaning  in  contextually  relative  ways, 

•  enable  humans  to  achieve  insights, 

•  automatically  infer  the  hypotheses  that  the  human  is  considering, 
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•  enable  people  to  access  others’  intuitions,  and 

•  present  information  in  relevant  ways  and  defined  in  terms  of  some  magically  derived  model  of  the  human 
subconscious  or  its  storehouse  of  tacit  knowledge. 

These  envisioned  capabilities  appear  to  be  good  things  to  have,  and  the  call  for  research  on  such  capabilities  might 
serve  to  throw  down  a  gauntlet  and  thereby  push  the  envelope  of  intelligent  systems.  But  we  see  in  various  funding 
opportunities  and  program  descriptions  little  actual  relationship  to  the  notion  of  sensemaking,  especially  to 
empiricalresearch  findings  from  the  field  of  naturalistic  decision  making.  This  essay  examines  sensemaking  from 
various  perspectives  to  see  if  we  can  separate  the  things  that  are  doable  from  the  things  that  seem  more  like  pie-in- 
the-sky. 


Klein,  G.,  Moon,  B.,  &  Hoffman,  R.  R.  (2007).  Making  Sense  of  Sensemaking  2:  A  Macrocognitive  Model.  IEEE 
Intelligent  Systems,  22-26. 
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Summary  abstract: 

In  our  first  essay  on  sensemaking,  1  we  discussed  various  possible  meanings  of  the  concept  and  debunked  some  of 
the  myths  that  seem  current  in  discussions  of  cognitive  work.  The  motivation  for  these  two  essays  is  to  question 
whether  it  makes  sense  to  envision  certain  kinds  of  intelligent  sensemaking  systems.  None  of  the  “verdicts”  we 
announced  in  the  first  essay  mean  that  intelligent  technologies  might  not  assist  people  in  sensemaking.  Indeed, 
intelligent  technologies  might  help;  they  just  won’t  be  the  sorts  of  technologies  that  people  seem  to  seek. 

Gary  Klein  and  his  colleagues  have  laid  out  a  theory  of  sensemaking  that  might  be  useful  for  intelligent  systems 
applications.2  It’s  a  general,  empirically  grounded  account  of  sensemaking  that  goes  significantly  beyond  the  myths 
and  puts  foiward  some  nonobvious,  testable  hypotheses  about  the  process. 

When  people  try  to  make  sense  of  events,  they  begin  with  some  perspective,  viewpoint,  or  framework — however 
minimal.  For  now,  let’s  use  a  metaphor  and  call  this  a  frame.  We  can  express  frames  in  various  meaningful  forms, 
including  stories,  maps,  organizational  diagrams,  or  scripts,  and  can  use  them  in  subsequent  and  parallel  processes. 
Even  though  frames  define  what  count  as  data,  they  themselves  actually  shape  the  data  (for  example,  a  house  fire 
will  be  perceived  differently  by  the  homeowner,  the  firefighters,  and  the  arson  investigators).  Furthermore,  frames 
change  as  we  acquire  data.  In  other  words,  this  is  a  twoway  street:  Frames  shape  and  define  the  relevant  data,  and 
data  mandate  that  frames  change  in  nontrivial  ways. 

Figure  1  shows  that  the  basic  sensemaking  act  is  dataframe  symbiosis.  The  figure  captures  a  number  of  sensemaking 
activities.  Sensemaking  can  involve  elaborating  the  frame  by  adding  details,  and  questioning  the  frame  and  doubting 
the  explanations  it  provides. 3  A  frame  functions  as  a  hypothesis  about  the  connections  among  data.  One  reaction  to 
doubt  is  to  explain  away  troublesome  data  and  preserve  the  frame. 4, 5  These  two  aspects,  elaborating  the  frame  and 
preserving  the  frame,  are  part  of  the  elaboration  cycle  of  sensemaking  (the  left  side  of  figure  1),  akin  to  Jean  Piaget’s 
notion  of  assimilation. 

Yet  another  sensemaking  cycle  is  to  reframe  (see  the  figure’s  right  side).  Here,  questioning  the  frame  leads  us  to 
reconsider — to  reject  the  initial  frame  and  seek  to  replace  it  with  a  better  one.  We  might  compare  alternative  frames 
to  determine  which  seems  most  accurate.  Or  we  might  simply  be  mystified  by  the  events.  The  sensemaking  activity 
here,  akin  to  Piaget’s  notion  of  accommodation,  is  to  find  some  sort  of  frame  that  plausibly  links  the  events  that  are 
being  explained. 

Each  of  these  aspects  of  sensemaking  has  its  own  dynamics,  strategies,  and  requirements.  Recognizing  a  frame  and 
recognizing  data  are  different  from  elaborating  a  frame  that  has  already  been  adopted,  and  this  is  different  from 
explaining  away  inconsistencies.  Different  still  are  the  reactions  to  questioning  a  frame — choosing  between 
alternative  frames  and  constructing  a  frame  where  none  exists. 

The  Data/Frame  Theory  posits  a  closed-loop  transition  sequence  between 

•  mental  model  formation  (which  is  backward  looking  and  explanatory),  and 

•  mental  simulation  (which  is  forward  looking  and  anticipatory). 

Think  of  the  simplest  transition  sequence  as  a  chain  of  closed  loops.  Each  loop  is  triggered  by  a  perceived  subevent, 
leading  to  an  effort  to  refine  the  existing  mental  model  (backward  looking)  and  an  effort  to  run  a  new  mental 
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simulation  (forward  looking).  You  can  construct  a  transition  sequence  retrospectively  to  generate  an  explanation  of 
how  events  and  subevents  unfolded,  or  prospectively  to  imagine  how  a  major  causal  factor  or  a  situational  mix  of 
factors  might  play  out.  For  illustration,  envision  a  transition  sequence  using  the  metaphor  of  billiards,  where  a  player 
would  anticipate  how  hitting  one  ball  would  lead  to  motion  in  a  second,  and  a  third,  to  the  shot’s  completion. 


Klein,  G.,  Pliske,  R.,  Crandall,  B.,  &  Woods,  D.  D.  (2005).  Problem  Detection.  Cognition,  Technology,  and  Work, 
7(1),  14-28. 
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Summary  abstract: 

Problem  detection  is  the  process  by  which  people  first  become  concerned  that  events  may  be  taking  an  unexpected 
and  undesirable  direction  that  potentially  requires  action.  Previous  accounts  [e.g.,  Cowan  (Acad  Manage  Rev 
1 1  (4):  763— 776,  1986)]  described  problem  detection  as  the  accumulation  of  discrepancies  until  a  threshold  was 
reached.  In  reviewing  incidents  taken  from  a  variety  of  natural  settings,  we  found  that  discrepancy  accumulation  did 
not  apply  to  the  incidents  we  reviewed,  because  (a)  cues  to  problems  may  be  subtle  and  context-dependent,  and  (b) 
what  counts  as  a  discrepancy  depends  on  the  problem-solver’s  experience  and  the  stance  taken  in  interpreting  the 
situation.  In  many  cases,  detecting  a  problem  is  equivalent  to  reconceptualizing  the  situation. 


Klein,  G.,  Ross,  K.  G.,  Moon,  B.  M.,  Klein,  D.  E.,  Hoffman,  R.  R.,  &  Hollnagel,  E.  (2003).  Macrocognition.  IEEE 
Intelligent  Systems,  18(3),  81-85. 
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Summary  abstract: 

If  we  engineer  complex  cognitive  systems  on  the  basis  of  mistaken  or  inappropriate  views  of  cognition,  we  can  wind 
up  designing  systems  that  degrade  performance  rather  than  improve  it.  The  results  stemming  from  the  application  of 
any  cognitive  systems  engineering  methodology  will  be  incomplete  unless  they  include  a  description  of  the 
cognition  that  is  needed  to  accomplish  the  work.  The  concept  of  macrocognition  is  a  way  of  describing  cognitive 
work  as  it  naturally  occurs. 


Klein,  G.,  Veinott,  E.  S.,  &  Sieck,  W.  R.  (2009).  Chapter  18.  Advancing  Our  Understanding  of  Batilefield  Decision 
Making.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter: 

Foundations  and  Technology.  Section  IV.  Communicating  Information  across  the  Team:  48hrBooks  by  Alion 
Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving 
Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision 
Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board 
Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue 
SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 
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The  process  of  command  and  control,  which  is  central  to  U.S.  Army  operations,  depends  heavily  on  cognitive 
functions  such  as  decision  making.  Commanders  and  staff  members  are  engaged  in  planning  and  replanning,  in 
interpreting  the  actions  of  adversaries,  in  re-programming  resources,  in  making  personnel  assignments,  and  in  many 
related  activities.  They  have  to  make  high  stakes  decisions  under  uncertainty  and  time  pressure.  In  order  to  support 
them  through  training,  organizational  design,  and  decision  support  systems,  it  is  essential  to  understand  how  they 
can  use  their  experience  to  make  effective  decisions  when  faced  with  complex  conditions. 

In  1989,  the  Army  Research  Institute  for  Behavioral  and  Social  Sciences  sponsored  the  first  conference  on 
Naturalistic  Decision  Making  (NDM,  Klein,  Orasanu,  Calderwood,  &  Zsambok,  1993).  The  NDM  framework  was 
designed  to  investigate  decision  making  under  difficult  conditions:  vague  goals,  uncertainty,  changing 
environmental  features,  high  stakes,  team  and  organizational  constraints  on  action  (Orasanu  &  Connolly,  1993).  It  is 
important  to  understand  decision  making  under  these  kinds  of  conditions  in  order  to  develop  effective  training 
programs,  information  technology  applications,  and  doctrine. 

One  ofthe  best-known  of  the  NDM  accounts  is  the  Recognition  Primed  Decision  (RPD)  model,  which  explains  how 
people  are  able  to  make  decisions  without  having  to  compare  options  (Klein,  1998;  Klein,  Calderwood,  &  Clinton- 
Cirocco,  1986).  The  RPD  model  posits  that  people  are  able  to  rely  heavily  on  the  first  option  they  generate  because 
their  experience  has  been  compiled  as  patterns.  A  pattern-matching  process  enables  them  to  rapidly  match  a 
situation  to  a  prototype  pattern  they  have  already  compiled,  which  lets  them  focus  their  attention  on  relevant  cues, 
determine  plausible  goals,  form  expectancies,  and  identify  promising  courses  of  action.  Therefore,  in  most  cases  the 
first  option  they  consider  will  be  effective.  They  only  have  to  generate  additional  options  when  their  first  option  is 
evaluated  as  unacceptable.  They  do  not  even  have  to  generate  several  options  in  order  to  make  a  comparative 
evaluation.  The  evaluation  is  conducted  through  a  process  of  mental  simulation  (which  De  Groot,  1946/1978, 
termed  "progressive  deepening"). 

The  RPD  model  has  been  tested.  The  initial  research  with  firefighters  has  been  extended  to  other  domains,  and  the 
dominant  finding  is  that  the  strategy  described  by  the  model  is  used  in  the  great  majority  of  situations-  between  80% 
and  90%  ofthe  time.  Rarely  do  decision  makers  contrast  alternative  options  using  a  common  set  of  evaluation 
criteria.  These  findings  have  been  replicated  by  different  research  groups.  Klein  (1998)  has  provided  a 
comprehensive  account  of  the  tests  and  replications  of  the  RPD  model. 

By  the  time  the  Army  Research  Laboratory  Advanced  Decision  Architectures  (ARL  ADA)  program  began  in  2001, 
the  NDM  framework  had  become  well  established  for  studying  decision  making  in  field  settings,  and  the  RPD 
model  had  become  widely  accepted  as  a  description  of  the  way  people  actually  make  decisions.  The  ARL  ADA 
program  has  been  extremely  valuable  for  moving  our  understanding  of  decision  making  forward.  It  directly  resulted 
in  advances  in  the  RPD  model,  and  in  the  extension  of  the  NDM  framework  to  cover  additional  cognitive  functions 
and  processes.  This  chapter  describes  how  the  ARLADA  program  enabled  Klein  Associates  to  make  important 
progress.  Other  ARLADA  contractors  have  also  made  important  progress,  particularly  in  modeling  recognitiona  I 
decision  making. 


Klein,  G.,  Wiggins,  S.  L.,  &  Lewis,  W.  R.  (2003).  Replanning  in  the  Army  Brigade  Command  Post.  In  In 
Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the 
Objective  Force.  April  29  -  May  1,  2003  (pp.  197-200).  College  Park,  MD:  U.S.  Army  Research  Laboratory. 

Key  Words: 
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Summary  abstract: 

Although  replanning  is  critical  for  effective  operations,  it  has  received  little  attention  from  the  research  community 
to  date.  Replanning  presents  a  commander  and  staff  with  a  unique  set  of  demands  and  challenges.  Preliminary 
studies  of  this  topic  have  identified  the  kinds  of  disruptions  that  require  replanning  and  indicated  the  barriers  to 
effective  replanning.  The  current  effort  intends  to  describe  the  replanning  process,  identify  its  macrocognitive 
requirements,  and  formulate  requirements  for  replanning  support  systems.  In  preparation  for  this  research,  a  number 
of  cognitive  challenges  to  effective  replanning  have  been  identified.  The  main  activities  of  the  effort  are  to  conduct 
observations  and  Cognitive  Task  Analysis  interviews  at  Army  exercises  involving  Future  Combat  System  and 
Objective  Force  dynamics.  Several  such  exercises  have  been  attended  and  preliminary  insights  are  presented. 
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Klein,  G.,  Woods,  D.  D.,  Bradshaw,  J.  M.,  Hoffman,  R.  R.,  &  Feltovich,  P.  J.  (2005).  Ten  Challenges  for  Making 
Automation  a  “Team  Player”  in  Joint  Human-Agent  Activity  IEEE  Intelligent  Systems(November/December),  91- 
95. 
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Summary  abstract: 

We  propose  10  challenges  for  making  automation  components  into  effective  “team  players”  when  they  interact  with 
people  in  significant  ways.  Our  analysis  is  based  on  some  of  the  principles  of  human-centered  computing  that  we 
have  developed  individually  and  jointly  over  the  years,  and  is  adapted  from  a  more  comprehensive  examination  of 
common  ground  and  coordination. 


Knoll,  G.,  Suri,  N.,  &  Bradshaw,  J.  M.  (2002).  Path-Based  Security  for  Mobile  Agents.  Electronic  Notes  in 
Theoretical  Computer  Science  (ENTCS)  issue  on  Security  of  Mobile  Multiagent  Systems  (SEMAS).  Elsevier 
Science,  63,  54-60. 
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Summary  abstract: 

As  mobile  agents  are  increasingly  adopted  in  intranets,  on  the  Internet,  and  on  computational  grids,  new  security 
concerns  become  increasingly  important.  Unlike  other  kinds  of  mobile  code,  such  as  applets,  which  are  pulled  a 
single  time  to  remote  systems  (single-hop),  mobile  agents  may  move  using  their  own  itinerary  through  a  series  of 
systems  (multiple-hop),  potentially  carrying  sensitive  information  with  them.  In  such  scenarios,  mobile  agents 
introduce  new  vulnerabilities:  hosts  are  at  the  mercy  of  malicious  (or  buggy)  agents  that  may  compromise  the 
integrity  of  the  host  execution  environment  and  agents  are  at  the  mercy  of  malicious  hosts  that  are  capable  of 
examining  or  modifying  the  code  or  data  of  the  agent,  or  presenting  a  false  environment  to  the  agent  (as  in  a 
masquerade  attack),  thereby  causing  the  agent  to  execute  incorrectly.  Widely  used  security  measures  such  as 
cryptography,  authentication,  and  access  control  do  not  address  these  kinds  of  problems. 

A  particularly  dangerous  form  of  agent  tampering  by  a  malicious  host  occurs  in  the  multiple-hop  scenario.  In  this 
situation,  an  agent  that  was  benign  (and  potentially  trusted)  to  begin  with  could  be  turned  into  a  malicious  agent. 
Since  the  agent  was  originally  a  trusted  agent,  a  subsequent  host  might  naively  grant  the  agent  higher  privileges, 
which  could  be  misused  by  the  now  malicious  agent.  Such  an  agent  could  cause  significantly  more  damage  to  the 
unsuspecting  host  than  a  malicious  agent  that  was  not  trusted  by  the  host  and  thereby  executed  with  greater 
protections. 

Some  solutions  have  been  developed  to  address  both  security  concerns:  protecting  hosts  from  malicious  agents  and 
protecting  agents  from  malicious  hosts.  Note  that  it  is  also  possible  for  a  malicious  agent  to  attack  another  agent. 
However,  given  that  a  malicious  host  has  complete  control  over  an  agent’s  execution  environment,  the  points  of 
attack  available  to  a  malicious  host  comprise  a  superset  of  the  points  of  attack  available  to  a  malicious  agent. 
Therefore,  the  problem  of  protecting  an  agent  from  a  malicious  agent  is  subsumed  by  the  problem  of  protecting  the 
agent  from  a  malicious  host. 

In  general,  the  solutions  to  protect  hosts  are  successful  (if  the  hosts  applies  the  necessary  security  mechanisms  when 
executing  agents).  However,  the  solutions  to  protect  agents  have  not  been  successful.  The  goal  of  this  effort  is  not  to 
solve  the  problem  of  protecting  an  agent  from  a  malicious  host,  but  to  help  a  host  determining  the  appropriate  trust 
level  of  a  visiting  agent  so  that  a  host  may  protect  itself  using  the  necessary  security  mechanisms.  Section  two  of  this 
paper  provides  a  survey  of  existing  solutions  to  mobile  agent  security.  The  following  section  briefly  describes  the 
NOMADS  mobile  agent  environment  where  our  proposed  security  solution  is  being  implemented. 
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Section  four  presents  the  proposed  solution  for  path-based  security  for  mobile  agents.  Section  five  concludes  with  a 
summary  and  a  discussion  of  future  work. 


Koopman,  P.,  &  Hoffman,  R.  R.  (2004).  Work-Arounds,  Make-Work,  and  Kludges.  IEEE:  Intelligent  Systems,  pp. 
70-75. 
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Summary  abstract: 

Paradigms  are  often  defined  partly  in  terms  of  what  they  are  not,  or  in  terms  of  what  they  are  reacting  against.  The 
paradigm  of  human-centered  computing  is  no  exception.  In  response  to  an  essay  in  the  Jan./Feb.  2002  Human- 
Centered  Computing  column  “The  State  of  Cognitive  Systems  Engineering,”!  we  had  a  lengthy  discussion  on  the 
question.  What  is  a  user-hostile  system?  The  following  quote  is  from  that  essay:  The  road  to  user-hostile  systems  is 
paved  with  user-centered  intentions  on  the  part  of  the  designers.  Even  smart,  clever,  well-intentioned  people  can 
build  devices  that  are  fragile  and  hostile,  devices  that  force  the  human  to  adapt  and  build  local  kludges  and  work¬ 
arounds.  Worse  still,  even  if  one  is  aware  of  this  trap,  one  will  still  fall  into  it. 

We  decided  that  the  terms  kludge  and  work-around,  and  also  the  related  concept  of  make -work,  have  yet  to  be 
clearly  defined  for  the  intelligent  systems  community.  Human-centered  systems  are  different  from  user-hostile 
systems  as  well  as  from  systems  based  on  a  designercentered  approach. 2  In  this  essay,  we  try  to  clarify  the  senses  of 
these  three  terms  and  suggest  ways  we  might  study  work-arounds,  make -work,  and  kludges  as  an  integral  part  of 
human-computer  systems — rather  than  as  embarrassing  necessities  that  are  best  swept  under  the  computing  research 
rug. 


Kurup,  U.,  &  Chandrasekaran,  B.  (2006).  Multi-Modal  Cognitive  Architectures:  A  Partial  Solution  to  the  Frame 
Problem.  In  In  Proceedings  of  2006  Cognitive  Science  Conference. 
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Summary  abstract: 

Since  its  definition  by  McCarthy  in  1969,  the  Frame  Problem  (FP)  has  been  one  of  the  more  heavily  debated 
problems  in  AI.  Part  of  the  debate  has  been  on  the  exact  definition  of  what  the  FP  really  is.  The  computational 
aspect  of  the  FP  can  be  thought  of  as  reasoning  about  what  changes  and  what  doesn’t  change  in  a  dynamic  world. 
The  “sleeping  dog  strategy”  is  considered  to  be  a  viable  solution  to  this  aspect  of  the  FP.  We  intend  to  show  that  this 
strategy  has  a  weakness  that  can  be  partially  solved  using  diagrammatic  reasoning,  under  certain  conditions.  A 
related  and  equally  important  problem,  called  the  Ramification  Problem,  is  to  be  able  to  reason  about  the  indirect 
effects  of  an  action  in  the  world.  Our  proposal  provides  a  more  efficient  solution  to  the  Ramification  Problem  when 
reasoning  about  spatial  relations.  To  illustrate  our  solution,  we  introduce  a  problem  solving  architecture  based  on 
Soar  that  is  augmented  with  a  diagrammatic  reasoning  component.  A  problem  state  in  this  augmented  Soar  is  bi- 
modal  in  nature,  one  part  being  symbolic  and  the  other  diagrammatic.  We  describe  its  use  in  certain  problems  and 
show  how  the  use  of  diagrams  can  handle  the  frame  and  ramification  problems  with  respect  to  spatial  relations. 


Kurup,  U.,  &  Chandrasekaran,  B.  (2007).  Modeling  Memories  of  Large-Scale  Space  Using  a  Bimodal  Cognitive 
Architecture.  In  In  Proceedings  of  2007  Soar  Workshop.  Ann  Arbor,  MI.  Also  In  Proceedings  of  Eighth 
International  Conference  on  Cognitive  Modeling  (ICCM  '07). 
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Summary  abstract: 

We  present  an  extension  to  biSoar,  a  bimodal  version  of  the  cognitive  architecture  Soar,  by  adding  a  bimodal 
version  of  chunking,  Soar’s  basic  learning  mechanism.  We  show  how  this  new  biSoar  is  a  useful  tool  in  modeling 
cognitive  phenomena  involving  spatial  or  diagrammatic  elements  by  applying  it  to  the  modeling  of  problem  solving 
involving  large-scale  space,  such  as  way-finding.  We  suggest  how  such  models  can  help  in  identifying  variables  to 
control  for  in  human  subject  experiments. 


Kurup,  U.,  &  Chandrasekaran,  B.  (2009).  A  Cognitive  Map  for  an  Artificial  Agent.  In  In  Proceedings  of  Artificial 
General  Intelligence  2009  Conference  in  Arlington,  VA. 

Key  Words: 
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Summary  abstract: 

We  show  how  a  general-purpose  cognitive  architecture  augmented  with  a  general  diagrammatic  component  can 
represent  and  reason  about  Large-scale  Space.  The  diagrammatic  component  allows  an  agent  built  in  this 
architecture  to  represent  information  both  symbolically  and  diagrammatically  as  appropriate.  Using  examples  we 
show  (a)  how  the  agent’s  bimodal  representation  captures  its  knowledge  about  large-scale  space  as  well  as  how  it 
learns  this  information  while  problem  solving  and  (b)  the  agent’s  flexibility  when  it  comes  to  using  learned 
information  and  incorporating  new  information  in  solving  problems  involving  large-scale  space. 


Laplante,  P.,  Hoffman,  R.  R.,  &  Klein,  G.  (2007).  Antipatterns  in  the  Creation  of  Intelligent  Systems.  IEEE 
Intelligent  Systems,  JANUARY/FEBRUARY  2007,  91-95. 
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Summary  abstract: 

A  design  pattern  is  a  named  problem-solution  pair  that  enables  large-scale  reuse  of  software  architectures  or  their 
components  (that  is,  interface  designs  1).  Ideally,  patterns  explicitly  capture  expert  knowledge,  design  trade-offs,  and 
design  rationale  and  make  these  lessons  learned  widely  available  for  off-the-shelf  use.  They  can  also  enhance 
developers’  vocabulary — for  example,  by  easing  the  transition  to  object-oriented  programming. 2  Conventionally, 
patterns  consist  of  four  elements:  a  name,  the  problem  to  be  solved,  the  solution  to  the  problem  (often  termed  the 
refactored  solution),  and  the  consequences  of  the  solution.  Numerous  sets  of  patterns  (collectively  known  as  pattern 
languages)  exist  for  software  design,  analysis,  management,  and  so  on;  a  Web  search  on  “pattern  language”  yields 
many  hits.  Shortly  after  the  notion  of  design  patterns  emerged,  practitioners  began  discussing  problem-solution  pairs 
in  which  the  solution  did  more  harm  than  good. 3, 4  These  have  come  to  be  known  as  antipatterns,  and  they  are  well 
known  in  the  design  and  management  communities. 


Laughery,  K.  R.,  Jr.,  Archer,  S.,  &  Corker,  K.  (2001).  Chapter  93.  Modeling  Human  Performance  in  Complex 
Systems.  In  G.  Salvendy  (Ed.),  Handbook  of  Industrial  Engineering  (3  ed.,  pp.  2409-2443).  New  York:  John  Wiley 
&  Sons,  Inc. 
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Summary  abstract: 

Over  the  past  few  decades,  human  factors  and  ergonomics  practitioners  have  increasingly  been  called  upon  early  in 
the  system  design  and  development  process.  Early  input  from  all  disciplines  results  in  better  and  more  integrated 
designs  as  well  as  lower  costs  than  if  one  or  more  disciplines  finds  that  changes  are  required  later.  Our  goal  as 
human  factors  and  ergonomics  practitioners  should  be  to  provide  substantive  and  well-supported  input  regarding  the 
human(s),  his  or  her  interaction(s)  with  the  system,  and  the  resulting  total  performance.  Furthermore,  we  should  be 
prepared  to  provide  this  input  from  the  earliest  stages  of  system  concept  development  and  then  throughout  the  entire 
system  or  product  life  cycle. 

To  meet  this  challenge,  many  human  factors  and  ergonomics  tools  and  technologies  have  evolved  over  the  years  to 
support  early  analysis  and  design.  Two  specific  types  of  technologies  are  design  guidance  (e.g.,  O'Hara  et  al.  1995; 
Boff  et  al.  1986)  and  high-fidelity  rapid  prototyping  of  user  interfaces  (e.g.,  Dahl  et  al.  1995).  Design  guidance 
technologies,  in  the  form  of  either  handbooks  or  computerized  decision  support  systems,  put  selected  portions  of  the 
human  factors  and  ergonomics  knowledge  base  at  the  fingertips  of  the  designer,  often  in  a  form  tailored  to  a 
particular  problem  such  as  nuclear  power  plant  design  or  UNIX  computer  interface  design.  However,  design  guides 
have  the  shortcoming  that  they  do  not  often  provide  methods  for  making  quantitative  trade-offs  in  system 
performance  as  a  function  of  design.  For  example,  design  guides  may  tell  us  that  a  high-resolution  color  display  will 
be  better  than  a  black-and-white  display,  and  they  may  even  tell  us  the  value  in  terms  of  increased  response  time  and 
reduced  error  rates.  However,  this  type  of  guidance  will  rarely  provide  good  insight  into  the  value  of  this  improved 
element  of  the  human's  performance  to  the  overall  system's  performance.  As  such,  design  guidance  has  limited  value 
for  providing  concrete  input  to  system-level  performance  prediction. 

Rapid  prototyping,  on  the  other  hand,  supports  analysis  of  how  a  specific  design  and  task  allocation  will  affect 
human  and  system-level  performance.  The  disadvantage  of  prototyping,  as  with  all  human-subjects  experimentation, 
is  that  it  can  be  costly.  In  particular,  prototypes  of  hardware-based  systems,  such  as  aircraft  and  machinery,  are  very 
expensive  to  develop,  particularly  at  early  design  stages  when  there  are  many  widely  divergent  design  concepts.  In 
spite  of  the  expense,  hardware  and  software  prototyping  are  important  tools  for  the  human  factors  practitioner,  and 
their  use  is  growing  in  virtually  every  application  area. 

While  these  technologies  are  valuable  to  the  human  factors  practitioner,  what  is  often  needed  is  an  integrating 
methodology  that  can  extrapolate  from  the  base  of  human  factors  and  ergonomics  data,  as  reflected  in  design  guides 
and  the  literature,  in  order  to  support  system-level  performance  predictions  as  a  function  of  design  alternatives.  This 
methodology  should  also  bind  with  rapid  prototyping  and  experimentation  in  a  mutually  supportive  and  iterative 
way.  As  has  become  the  case  in  many  engineering  disciplines,  a  prime  candidate  for  this  integrating  methodology  is 
computer  modeling  and  simulation. 

Computer  modeling  of  human  behavior  and  performance  is  not  a  new  endeavor.  Computer  models  of  complex 
cognitive  behavior  have  been  around  for  over  20  years  (e.g.,  Newell  and  Simon  1972)  and  tools  for  computer 
modeling  of  task-level  performance  have  been  available  since  the  1970s  (e.g.,  Wortman  et  al.  1978).  However,  two 
things  have  changed  appreciably  in  the  past  decade  that  promote  the  use  of  computer  modeling  and  simulation  of 
human  performance  as  a  standard  tool  for  the  practitioner.  First  is  the  rapid  increase  in  computer  power  and  the 
associated  development  of  easierto-  use  modeling  tools.  Individuals  with  an  interest  in  predicting  human 
performance  through  simulation  can  select  from  a  variety  of  computer-based  tools  (for  a  comprehensive  list  of  these 
tools,  see  McMillan  et  al.  1989).  Second  is  the  increased  focus  by  the  research  community  on  the  development  of 
predictive  models  of  human  performance  rather  than  simply  descriptive  models.  For  example,  the  GOMS  model 
(Gray  et  al.  1993)  represents  the  integration  of  research  into  a  model  for  making  predictions  of  how  humans  will 
perform  in  a  realistic  task  environment.  Another  example  is  the  research  in  cognitive  workload  that  has  been 
represented  as  computer  algorithms  (e.g.,  McCracken  and  Aldrich  1984;  Farmer  et  al.  1995).  Given  a  description  of 
the  tasks  and  equipment  with  which  humans  are  engaged,  these  algorithms  support  assessment  of  when  workload- 
related  performance  problems  are  likely  to  occur  and  often  include  identification  of  the  quantitative  impact  of  those 
problems  on  overall  system  performance  (Hahler  et  al.  1991).  These  algorithms  are  particularly  useful  when 
embedded  as  key  components  in  computer  simulation  models  of  the  tasks,  and  the  environment. 

Perhaps  the  most  powerful  aspect  of  computer  modeling  and  simulation  is  that  it  provides  a  method  through  which 
the  human  factors  and  ergonomics  team  can  step  up  to  the  table  with  the  other  engineering  disciplines  that  also  rely 
increasingly  on  quantitative  computer  models.  What  we  will  discuss  in  this  chapter  are  the  methods  through  which 
the  human  factors  and  ergonomics  community  can  contribute  early  to  system  design  tradeoff  decisions. 
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Laux,  L.,  Laughery,  R.,  Endsley,  M.,  &  Strub,  M.  (2003).  Designing  Systems  around  Soldier  Decision  Making  and 
Situation  Awareness:  Improving  C4isr  Process.  In  In  Proceedings  of  Army  Science  Conference.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  most  military  systems  today  enormous  amounts  of  data  are  available  to  be  presented  to  the  soldier  and  the  volume 
of  available  C4ISR  data  is  continually  increasing.  Multiply  this  by  the  large  numbers  of  formats  (technologies) 
available  for  presenting  these  data  and  the  decisions  the  interface  and  interaction  designer  must  make  become 
unmanageable  without  direction.  This  paper  describes  a  methodology  for  managing  this  data  overload  problem 
through  a  structured  process  for  maximizing  solider  situation  awareness  to  improve  decision  making  in  C4ISR. 


Laux,  L.,  McDermott,  P.,  Keller,  J.,  &  Laughery,  R.  (2006).  Visualization,  Cognitive  Task  Analysis,  and  Decision- 
Centered  Design.  In  Presented  at  United  States  Strategic  Command  (USSTRATCOM)  Command  and  Control  (C2) 
Technology  Summit:  Data  Visualization  and  Data  Discovery/Access  for  Net-centric  C2  Architectures,  Bellevue, 
Nebraska. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Communication,  Information,  and  Visualization 

A  user  interface  is  a  communication  device.  The  Graphical  User  Interface  (GUI)  of  a  military  application  is  a 
primarily  visual  (sometimes  auditory)  communication  device  intended  to  provide  useful  information  and  facilitate 
decision-making.  In  Information  Theory  terms,  a  GUI  component  such  as  a  graphical  display  or  text  message  only 
provides  information  when  it  communicates  something  the  user  doesn’t  already  know  and  reduces  uncertainty.  The 
goal  of  creating/presenting  visualizations  is  to  increase  the  amount  of  information  in  a  display  or  series  of  displays 
while  concomitantly  reducing  the  perceptual  and  cognitive  load  on  the  decision-maker. 

Two  trends  suggest  that  we  need  to  develop  methods,  such  as  visualizations,  to  reduce  the  perceptual  and  cognitive 
load  on  decision-makers  who  are  receiving  intelligence  and  other  forms  of  information  electronically: 

1.  Systems  are  becoming  more  automated  and  the  role  of  decision-maker  is  shifting  to  that  of  system  monitor 
(rather  than  “hands-on  operator”).  As  systems  become  more  automated,  the  types  of  decisions  that  must  be  made 
and  the  complexity  of  the  data  will  make  it  more  difficult  for  the  decision-makers  who  monitor  these  systems  to 
maintain  situation  awareness  because  they  will  not  be  engaged  and  in-the-loop.  Visualizations,  such  as  dynamic 
graphs,  can  be  used  to  create  pertinent  status  information  and  reduce  the  load  associated  with  processing  multiple 
sources  of  data. 

2.  Sensor  and  other  technology  advances  provide  us  with  more  and  more  data/time  period  and  these  advances  are 
expected  to  continue  over  the  foreseeable  future,  meaning  that  the  amount  of  data  we  have  available  to  present  to 
users  is  overwhelming  -  operators  simply  cannot  efficiently  process  the  amount  of  data  we  currently  have  to  present 
them  unless  we  can  “condense”  the  data  and  turn  it  into  meaningful  information  that  can  be  quickly  understood  and 
assimilated.  Visualizations  can  make  multi-dimensional  complex  data  readily  understood. 


Lebiere,  C„  Biefeld,  E.,  Archer,  R.,  Archer,  S.,  Allender,  L.,  &  Kelley,  T.  D.  (2002).  IMPRINT/ACT-R:  Integration 
of  a  Task  Network  Modeling  Architecture  with  a  Cognitive  Architecture  and  Its  Application  to  Human  Error 
Modeling.  In  In  Proceedings  of  Military,  Government,  and  Aerospace  Simulation  (MGA  2002)  Conference.  Michael 
Chinni,  (Ed.),  pp.  13-18. 
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This  paper  describes  ongoing  efforts  to  integrate  IMPRINT  (IMproved  Performance  Research  INtegration  Tool),  a 
task  network  modeling  architecture  with  ACT-R  (Adaptive  Character  of  Thought  —  Rational),  a  hybrid  cognitive 
architecture.  IMPRINT  consists  of  a  set  of  automated  aids  to  conduct  human  performance  analyses  built  on  top  of 
the  Micro  Saint  task  network  modeling  environment.  ACT-R  combines  a  goaldirected  production  system  with  a 
subsymbolic  activation  calculus  that  tunes  itself  to  the  structure  of  the  environment  using  Bayesian  learning 
mechanisms.  Because  ACT-R  and  IMPRINT  were  targeted  at  different  behavioral  levels,  they  perfectly  complement 
each  other.  IMPRINT  is  focused  on  the  task  level,  how  high-level  functions  break  down  into  smaller-scale  tasks  and 
the  logic  by  which  those  tasks  follow  each  other  to  accomplish  those  functions.  ACT-R  is  targeted  at  the  OatomicO 
level  of  thought,  the  individual  cognitive,  perceptual  and  motor  acts  that  take  place  at  the  subsecond  level.  Goals  in 
ACT-R  correspond  directly  to  tasks  in  IMPRINT,  providing  a  natural  integration  level.  Certain  tasks  in  an 
IMPRINT  task  network  can  be  implemented  as  ACT-R  models,  combining  the  cognitive  accuracy  of  a  cognitive 
architecture  with  the  tractability  and  ease  of  design  of  task  networks. 

A  hybrid  IMPRINT/ACT-R  model  works  as  follows.  The  IMPRINT  model  specifies  the  network  of  tasks  and 
includes  the  definition  of  how  higher-order  functions  are  decomposed  into  tasks  and  the  logic  by  which  these  tasks 
are  composed  together.  For  certain  tasks,  IMPRINT  sends  to  ACT-R  over  a  Component  Object  Model  (COM)  link 
the  state  of  variables  providing  a  detailed  description  of  that  task.  ACT-R  then  creates  a  goal  corresponding  to  that 
task,  with  the  components  of  the  goal  set  to  the  description  of  the  task.  The  ACTR  model  for  that  goal  is  then  run, 
producing  detailed  cognitive  predictions  including  latency  of  the  run,  whether  an  error  occurred,  etc.  Those  results 
are  then  passed  back  over  the  same  COM  link  to  IMPRINT,  which  uses  them  as  parameters  of  the  task  to  advance 
the  task  network  model.  We  describe  an  application  of  this  hybrid  modeling  to  the  prediction  of  human  errors  that 
lead  to  runway  incursions.  Finally,  we  discuss  future  extensions  of  our  work,  including  the  use  of  a  standardized 
High  Level  Architecture  (HLA)  link  to  handle  communications  between  IMPRINT  and  ACT-R  and  the  extension  of 
the  task  parameters  exchanged  to  include  workload  predictions. 


Lebiere,  C.,  Gonzalez,  C.,  &  Martin,  M.  (2008).  Instance-Based  Decision  Making  Model  of  Repeated  Binary 
Choice.  In  In  Proceedings  of  Lewis,  Polk,  &  Laird  (Eds.),  8th  International  Conference  on  Cognitive  Modeling 
(ICCM)  (pp.  77-82).  Ann  Arbor,  MI. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Learning;  dynamic  decision  making;  RELACS;  memory;  cognitive  architectures;  ACT-R. 

Summary  abstract: 

We  describe  an  instance -based  model  of  decision-making  for  repeated  binary  choice.  The  model  provides  an 
accurate  account  of  existing  data  of  aggregate  choice  probabilities  and  individual  differences,  as  well  as  newly 
collected  data  on  learning  and  choice  interdependency.  In  particular,  the  model  provides  a  general  emergent  account 
of  the  risk  aversion  effect  that  does  not  require  any  metacognitive  assumptions.  Advantages  of  the  model  include  its 
simplicity,  its  compatibility  with  previous  models  of  choice  and  dynamic  control,  and  the  strong  constraints  it 
inherits  from  the  underlying  cognitive  architecture. 


Lebiere,  C.,  Gonzalez,  C.,  &  Warwick,  W.  (2009).  A  Comparative  Approach  to  Understanding  General  Intelligence: 
Predicting  Cognitive  Performance  in  an  Open-Ended  Dynamic  Task.  In  In  Proceedings  of  Second  Conference  on 
Artificial  General  Intelligence.  Atlantis  Press,  Amestrdam-Paris,  pp. 103-107. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  evaluation  of  an  AGI  system  can  take  many  forms.  There  is  a  long  tradition  in  Artificial  Intelligence  (AI)  of 
competitions  focused  on  key  challenges.  A  similar,  but  less  celebrated  trend  has  emerged  in  computational  cognitive 
modeling,  that  of  model  comparison.  As  with  AI  competitions,  model  comparisons  invite  the  development  of 
different  computational  cognitive  models  on  a  well-defined  task.  However,  unlike  AI  where  the  goal  is  to  provide 
the  maximum  level  of  functionality  up  to  and  exceeding  human  capabilities,  the  goal  of  model  comparisons  is  to 
simulate  human  performance.  Usually,  goodness-of-fit  measures  are  calculated  for  the  various  models.  Also  unlike 
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AI  competitions  where  the  best  performer  is  declared  the  winner,  model  comparisons  center  on  understanding  in 
some  detail  how  the  different  modeling  “architectures”  have  been  applied  to  the  common  task.  In  this  paper  we 
announce  a  new  model  comparison  effort  that  will  illuminate  the  general  features  of  cognitive  architectures  as  they 
are  applied  to  control  problems  in  dynamic  environments.  We  begin  by  briefly  describing  the  task  to  be  modeled, 
our  motivation  for  selecting  that  task  and  what  we  expect  the  comparison  to  reveal.  Next,  we  describe  the 
programmatic  details  of  the  comparison,  including  a  quick  survey  of  the  requirements  for  accessing,  downloading 
and  connecting  different  models  to  the  simulated  task  environment.  We  conclude  with  remarks  on  the  general  value 
in  this  and  other  model  comparisons  for  advancing  the  science  of  AGI  development. 


Lebiere,  C.,  Gonzalez,  C.,  &  Warwick,  W.  (2009).  Convergence  and  Constraints  Revealed  in  a  Qualitative  Model 
Comparison.  Journal  of  Cognitive  Engineering  and  Decision  Making,  3(2,  Summer  2009),  13 1-155. 
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Summary  abstract: 

We  contrasted  and  compared  independently  developed  computational  models  of  human  performance  in  a  common 
dynamic  decision-making  task.  The  task,  called  dynamic  stocks  and  flows,  is  simple  and  tractable  enough  for 
laboratory  experiments  yet  exhibits  many  characteristics  of  macrocognition.  A  macrocognitive  model  was  developed 
using  a  computational  instantiation  of  recognition-primed  decision  making.  A  microcognitive  model  was  developed 
using  the  Adaptive  Control  of  Thought  -  Rational  (ACT-R)  cognitive  architecture.  Both  models  followed  an 
instancebased  learning  paradigm  and  displayed  striking  similarities,  including  their  constraints,  limitations,  and  the 
key  breakthrough  that  enabled  satisfactory  (though  still  short  of  human-like)  performance,  suggesting  the  emergence 
of  a  general  design  pattern.  On  the  basis  of  this  comparison  we  argue  that  although  some  substantive  differences 
remain,  microcognitive  and  macrocognitive  approaches  provide  complementary  rather  than  contradictory  accounts 
of  human  behavior. 


Lee,  D.-S.,  Woods,  D.  D.,  &  Kidwell,  D.  (2006).  Escaping  the  Design  Traps  of  Creeping  Featurism:  Introducing  a 
Fitness  Management  Strategy.  In  In  Proceedings  of  Usability  Professionals'  Association  Annual  Conference, 
Broomfield,  CO. 
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Summary  abstract: 

Creeping  featurism  is  a  process  that  produces  complexity.  This  paper  describes  ways  to  prevent  or  escape  from 
creeping  featurism.  We  discuss  limitations  of  current  methods  of  user  interface  design  and  usability  engineering,  and 
propose  a  fitness  management  strategy  to  detect  feature  creep  and  to  aid  the  search  for  innovations. 


Lockett,  J.  F.,  Ill,  &  Archer,  S.  (2009).  Chapter  51.  Impact  of  Digital  Human  Modeling  on  Military  Human-Systems 
Integration  and  Impact  of  the  Military  on  Digital  Human  Modeling.  In  V.G.  Duffy  (Ed.),  Handbook  of  Digital 
Human  Modeling.  6000  Broken  Sound  Parkway  NW,  Suite  300,  Boca  Raton,  FL  33487-2742:  CRC  Press.  Taylor 
&  Francis  Group. 
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Task  Models 

Cognitive  Models 

Information  Flow  Models 

Integrated  and  Hybrid  Models  and  Tools 

Analysis  Approach 

Application  Issues 

Case  Studies 

Number  of  Operators  and  Workload 
Light  Helicopter  Experimental  (LHX)  and  NBC 
Fox  Operator  Workload  Analyses 
The  Future  Combat  System  (FCS) 

Brigade  Staffing 

Information  Flow 

Command  and  Control  Case  Study 

CART  Case  Study 

Maintenance  Manpower 

JBPDS  Case  Study 

Decision  Making 

GRBIL  Case  Study 

Workspace  Analysis  Using  Human 

Figure  Models 

Future  Combat  System  (FCS) 

A  Common  Approach 

Advanced  Amphibious  Assault  Vehicle 

Comanche  Helicopter 

Stryker 

Army  Airborne  Command  and  Control  System  (A2C2S) 

Future  Challenges  and  Development 

Summary  abstract: 

Military  organizations  have  driven  much  of  the  research  and  development  in  ergonomics  and  huma  1\  factors 
engineering  and  as  a  result  have  realized  significant  benefits.  During  the  past  three  decades,  these  organizations 
made  significant  investments  in  development  of  digital  human  models  as  tools  10  enable  human  systems  integration. 
Use  of  these  models  has  resulted  in  earlier  identification  of  usabil  ity  issues,  increased  opportunity  to  resolve  those 
issues  less  expensively,  and  more  effective  and  sati.,r  systems.  In  this  chapter,  we  review  these  developments  and 
discuss  their  application  via  case  studies  Although  military  investment  is  certainly  not  limited  to  the  United  States 
Department  of  Defense  (DoD),  given  the  basis  of  our  experience,  this  is  our  focus.  Additional  discussion  of  these 
develop  •  ments  and  human  systems  integration  can  be  found  in,  for  example,  Booher  (1990,  2003)  and  Salvendy 
(2006). 

Digital  human  models  represent  a  wide  range  of  human  physical,  mental,  and  social  attributes  and  behaviors.  Many 
of  the  models  and  the  analysis  tools  in  which  they  have  been  embedded  characterize  multiple  attributes  and  so  are 
not  easily  categorized.  However,  the  following  categories  are  useful  in  describing  the  predominant  attributes 
represented  by  the  models  most  commonly  used  for  human  sys  terns  integration  (HSI).  They  are  human  figure 
models,  task  models,  cognitive  models,  information  flow  models,  and  integrated  or  hybrid  models. 


Lott,  J.,  Bradshaw,  J.  M.,  Uszok,  A.,  &  Jeffers,  R.  (2004).  Using  KAoS  Policy  and  Domain  Services  within 
Cougaar.  In  In  Proceedings  of  Open  Cougaar  Conference  2004.  New  York  City,  NY. 

Key  Words: 
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Summary  abstract: 

KAoS  policy  and  domain  management  services  allow  for  the  specification,  management,  conflict  resolution,  and 
enforcement  of  policies  represented  in  OWL  within  contexts  established  by  domains.  We  discuss  the  application  of 
KAoS  services  in  providing  policy  management  for  robustness  and  survivability  in  the  context  of  the  DARPA 
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UltraLog  program,  a  large-scale  distributed  agent-based  system  running  on  the  Cougaar  agent  infrastructure.  Over 
the  course  of  the  program  we  were  able  to  demonstrate  that  a  semantically-rich  policy  system  could  provide 
exceptional  performance  and  responsiveness  even  under  very  demanding  conditions 


Luck,  J.,  McDermott,  P.  L.,  Allender,  L.,  &  Fisher,  A.  (2006).  Advantages  of  Co-Location  for  Effective  Human  to 
Human  Communication  of  Information  Provided  by  an  Unmanned  Vehicle.  In  In  Proceedings  of  50th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society,  San  Francisco,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Much  of  the  research  on  unmanned-vehicles  (UVs)  focuses  on  technology  or  interface  design.  This  study 
investigated  how  to  best  support  effective  communication  between  the  UV  operator  and  the  Soldier  in  the  field  using 
UV-provided  information  to  complete  a  mission.  In  a  previous  study  investigating  the  impact  of  different  team 
configurations  and  the  utility  of  supporting  communication  technologies,  our  team  found  co-location  of  team 
members  to  be  beneficial  (McDermott  et  al.,  2005).  In  this  experiment  we  investigate  what  aspects  of  colocation  are 
key  to  successful  team  performance:  Is  face-to-face  communication  vital  compared  to  voice-only  when  team 
members  are  distributed?  Is  the  ability  of  the  UV  operator  to  see  what  the  Soldier  performing  the  mission  can  see 
critical?  We  also  seek  additional  insight  to  inconclusive  results  from  the  first  study  regarding  the  utility  of  image 
transmission  and  access  to  an  electronic  map  displaying  both  the  UV  and  Soldier  locations. 


Luck,  J.  P.,  McDermott,  P.  L.,  &  Allender,  L.  (2005).  The  Communication  of  UAV-Provided  Information.  In  In 
Proceedings  of  Annual  Human  Factors  of  UAVs  workshop.  Mesa,  AZ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Unmanned  Aerial  Vehicles  (UAV)  have  enormous  potential  to  help  infantry  soldiers  navigate  safely  by  providing 
information  about  enemy  locations  and  other  obstacles.  Our  research  focuses  on  real-time  communication  between  a 
person  monitoring  UAV-provided  information  and  a  soldier  in  the  field  using  the  information  to  execute  a  mission. 
One  practical  consideration  is  the  location  of  the  operator  monitoring  the  UAV.  Can  this  “Information  Manager” 
(IM)  be  positioned  rearward  at  a  command  center  or  should  the  IM  be  co-located  with  the  soldiers  in  the  field? 

There  are  potential  advantages  and  disadvantages  to  either  location.  A  co-located  IM  has  excellent  and  immediate 
awareness  of  the  situation  in  the  field  -  what  the  soldiers  are  looking  at,  where  they  are  located,  and  subtle  cues  such 
as  the  attitude  of  civilians  on  the  street.  Likewise,  it  should  be  easier  for  a  co-located  IM  to  discern  when  his  unit 
leader  is  confused  or  misunderstood  UAV-provided  information.  The  disadvantage  of  being  co-located  is  the  high 
workload  demands  of  monitoring  the  information  provided  by  the  UAV,  reconciling  that  information  to  the  unit’s 
current  location,  and  performing  necessary  infantry  tasks  (i.e.,  move,  use  cover  and  concealment,  defend  the  unit  if 
necessary,  etc).  A  rearward  located  IM  does  not  have  these  workload  and  safety  issues  but  is  less  aware  of  the 
situation  in  the  field.  Can  technological  advances  such  as  GPS  tracking  be  used  to  increase  the  IM’s  “immersion”  in 
the  infantry  unit’s  situation?  This  study  investigated  the  implications  of  three  team  configurations  on  performance:  a 
co-located  IM,  are  rearward  IM,  and  a  rearward  IM  with  access  to  a  digital  map  updated  in  real-time  with  the 
soldier’s  and  UAV’s  location. 

An  experiment  was  performed  using  the  video  game  Raven  Shield  as  a  battlefield  simulator.  U.S.  Marine  Coips 
participants  operated  in  two-person  teams  to  conduct  move-to-objective  scenarios  similar  to  “Black-Hawk  Down.” 

A  friendly  helicopter  has  crashed  in  enemy  territory  and  a  Rescue  Mission  (RM)  player  is  tasked  to  navigate  to  the 
downed  pilot  as  quickly  as  possible  without  being  detected  by  enemy  forces.  The  IM  uses  a  UAV  to  scout  a  route 
and  guide  the  RM  to  the  crash  site  along  a  route  that  avoids  the  enemy.  Quantitative  performance  measures  included 
the  time  to  complete  the  mission  and  the  number  of  detections  by  the  enemy.  This  poster  will  concentrate  on 
analysis  of  the  post-mission  questionnaires  and  verbal  protocols  between  the  IM  and  RM.  The  questionnaire  asked 
participants  to  rate  the  difficulty  of  communicating  locations,  usefulness  of  UAV-provided  information,  and  overall 
situation  awareness.  The  verbal  protocols  were  analyzed  to  understand  the  types  of  mistakes  made  with  the  UAV, 
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the  mistakes  averted  because  of  the  UAV,  how  the  asset  was  used  (i.e.,  to  provide  a  high-level  overhead  view  or  a 
street  level  view),  the  push  and  pull  of  information  between  the  team  members,  and  the  types  of  commands  given  to 
the  RM.  The  implications  for  UAV  team  configurations  will  be  discussed. 


Luck,  J.  P.,  McDermott,  P.  L.,  &  Allender,  L.  (2006).  An  Investigation  of  Situational  Awareness  in  Real  World 
Control  of  Robotic  Assets  with  Communication  Latency.  In  In  Proceedings  of  the  3rd  Annual  Human  Factors  of 
Unmanned  Aerial  Vehicles  Workshop.  Mesa,  AZ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Unmanned  vehicles  (UVs)  are  already  being  used  in  a  variety  of  applications,  including  the  military  battlefield.  The 
applications  will  only  increase  as  robotic  technology,  such  as  autonomy,  continues  to  advance.  As  the  technology 
advances,  the  number  of  applications  will  increase,  as  will  the  demands  on  the  associated  network.  Limited 
bandwidth,  extended  transmission  ranges,  and  high  network  traffic  can  all  result  in  communication  latency.  A 
previous  publication  summarized  the  effects  on  operator  performance  from  two  experiments  investigating  the  effects 
of  communication  latency  when  controlling  a  robot  (Luck,  2006).  This  paper  will  detail  the  effects  on  operator 
Situational  Awareness  (SA)  from  the  same  two  experiments. 

The  experimental  task  was  a  simulated  MOUT  reconnaissance  in  which  participants  drove  an  UV  through  four 
courses.  Participants  were  given  a  map  depicting  the  route  and  the  location  of  reconnaissance  “spins”.  At  three  pre¬ 
determined  points  the  participants  were  instructed  to  stop,  the  map  was  hidden,  and  participants  attempted  to  mark 
the  UV’s  location  and  to  retrace  the  route  of  the  UV  as  accurately  as  possible,  including  deviations  from  the 
intended  route.  Participants  also  provided  subjective  ratings  of  SA  after  each  mission  and  filled  out  a  subjective 
questionnaire  at  the  conclusion  of  the  experiment. 

One  current  thrust  in  robotics  aims  to  increase  the  Level  Of  Automation  (LOA)  of  robots.  The  rationale  is  that, 
reducing  the  amount  of  control  required  of  the  operator  will  extend  the  utility  of  the  robot  and  simultaneously  reduce 
operator  workload.  The  results  from  this  study  indicate  that  increasing  the  LOA  will  also  decrease  the  impact  of 
latency  duration  on  operator  SA.  The  number  of  stoppages  to  gain  SA  and  the  number  of  SA  marking  errors  (i.e., 
incorrectly  marking  the  UV  location,  rooms  entered,  and  location  of  driving  errors)  were  all  significantly  lower 
under  the  higher  compared  to  the  lower  LOAs  tested. 

The  duration  of  latency  also  impacted  operator  SA.  No  significant  effects  on  the  ability  to  guide  the  robot  along  the 
proper  route  were  found.  However,  significantly  more  marking  errors  were  made  with  short  delays  than  long 
delays.  This  counter-intuitive  result  most  likely  indicates  that  the  higher  level  of  concentration  required  with  longer 
delays  aids  operators  in  maintaining  SA. 

Communication  latency  can  also  arise  in  different  forms.  Latency  duration  may  be  constant,  as  with  hardware  or 
path  induced  delays  (ex:  satellite -based  communication),  or  may  fluctuate  due  to  varying  traffic  or  routing 
conditions  (ex:  typical  internet  based  communication).  Significant  effects  on  control  performance  were  found  for 
varying  latency  (Luck,  2006),  however  very  few  effects  are  seen  for  SA.  While  subjective  ratings  indicate  more 
time  was  spent  on  maintaining  SA  under  variable  latency,  there  was  only  one  measured  effect  on  SA:  Participants 
made  more  incorrect  markings  of  drive  error  locations  in  variable  latency.  Accordingly  varying  latency  has  little 
effect  on  overall  operator  SA,  but  may  require  more  effort  and  have  a  negative  effect  on  an  operators  ability  to 
remember  small  events  while  en  route  to  a  known  goal. 

Latency  conditions  may  also  differ  between  commands  sent  from  the  user  to  the  robot  (U2R)  and  feedback  sent 
from  the  robot  to  the  user  (R2U).  Perceived  difficulty  in  maintaining  SA  was  much  larger  when  feedback  was 
delayed  then  when  control  signals  were  delayed.  Participants  also  made  more  navigation  mistakes  in  the  R2U 
condition,  however  no  differences  in  marking  mistakes  were  found. 

Before  making  decisions  about  both  the  use  and  design  of  UV  interfaces  and  systems,  it  is  important  to  understand 
the  impact  of  all  of  these  factors  on  an  operator’s  ability  to  maintain  awareness  of  the  situation. 


Luck,  J.  P.,  McDermott,  P.  L.,  Allender,  L.,  &  Russell,  D.  C.  (2006).  An  Investigation  of  Real  World  Control  of 
Robotic  Assets  under  Communication  Latency.  In  In  Proceedings  of  ACM  Human  Robot  Interaction  Conference, 
Salt  Lake  City,  UT. 
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Categories  and  Subject  Descriptors 

J.4  [Social  and  Behavioral  Science];  J.7  [Computers  in  Other 
Systems],  Military;  Human  Factors. 

General  Terms 

Performance,  Experimentation,  and  Human  Factors. 

Keywords 

Latency,  Delay,  Communication,  Teleoperation,  Robotics, 

Control,  and  Level  of  Automation 

Summary  abstract: 

Robots  are  already  being  used  in  a  variety  of  applications,  including  the  military  battlefield.  As  robotic  technology 
continues  to  advance,  those  applications  will  increase,  as  will  the  demands  on  the  associated  network 
communication  links.  Two  experiments  investigated  the  effects  of  communication  latency  on  the  control  of  a  robot 
across  four  Levels  Of  Automation  (LOAs),  (1)  full  teleoperation,  (2)  guarded  teleoperation,  (3)  autonomous  obstacle 
avoidance,  and  (4)  full  autonomy.  Latency  parameters  studied  included  latency  duration,  latency  variability,  and  the 
“direction”  in  which  the  latency  occurs,  that  is  from  user-to-robot  or  from  robot-to-user.  The  results  indicate  that  the 
higher  the  LOA,  the  better  the  performance  in  terms  of  both  time  and  number  of  errors  made,  and  also  the  more 
resistant  to  the  degrading  effects  of  latency.  Subjective  reports  confirmed  these  findings.  Implications  of  constant  vs. 
variable-latency,  user-to-robot  vs.  robot-to-user  latency,  and  latency  duration  are  also  discussed. 


Malkin,  R.  G.,  &  Waibel,  A.  (2005).  Classifying  User  Environment  for  Mobile  Applications  Using  Linear 
Autoencoding  of  Ambient  Audio.  In  In  Proceedings  of  ICASSP-05.  Philadelphia,  PA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Many  mobile  devices  and  applications  can  act  in  contextsensitive  ways,  but  rely  on  explicit  human  action  for 
context  awareness.  It  would  be  preferable  if  our  devices  were  able  to  attain  context  awareness  without  human 
intervention.  One  important  aspect  of  user  context  is  environment.  We  present  a  novel  method  for  classifying 
environment  types  based  on  acoustic  signals.  This  method  makes  use  of  linear  autoencoding  neural  networks,  and  is 
motivated  by  the  observation  that  biological  coding  systems  seem  to  be  heavily  influenced  by  the  statistics  of  their 
environments.  We  show  that  the  autoencoder  method  achieved  a  lower  error  rate  than  a  standard  gaussian  mixture 
model  on  a  representative  sample  task,  and  that  a  linear  combination  of  autoencoders  and  GMMs  yielded  better 
performance  than  either  alone. 


Martin,  M.  J.,  &  Foltz,  P.  W.  (2004).  Automated  Team  Discourse  Annotation  and  Performance  Prediction  Using 
Lsa.  In  In  Proceedings  of  Human  Language  Technology  conference  /  North  American  chapter  of  the  Association  for 
Computational  Linguistics  Annual  Meeting  HLT/NAACL  2004. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

We  describe  two  approaches  to  analyzing  and  tagging  team  discourse  using  Latent  Semantic  Analysis  (LSA)  to 
predict  team  performance.  The  first  approach  automatically  categorizes  the  contents  of  each  statement  made  by  each 
of  the  three  team  members  using  an  established  set  of  tags.  Performance  predicting  the  tags  automatically  was  15% 
below  human  agreement.  These  tagged  statements  are  then  used  to  predict  team  performance.  The  second  approach 
measures  the  semantic  content  of  the  dialogue  of  the  team  as  a  whole  and  accurately  predicts  the  team.s  performance 
on  a  simulated  military  mission. 
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Martin,  M.  K.,  Gonzalez,  C.,  &  Lebiere,  C.  (2004).  Learning  to  Make  Decisions  in  Dynamic  Environments:  ACT-R 
Plays  the  Beer  Game.  In  In  Proceedings  of  6th  International  Conference  on  Cognitive  Modeling.  Pittsburgh,  PA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Sterman  (1989)  proposed  that  decision  makers  misperceive  the  feedback  provided  by  dynamically  complex 
environments,  and  questioned  whether  people  can  learn  to  make  effective  decisions  in  such  environments.  We 
provide  empirical  evidence  of  learning  in  a  well-known  dynamic  environment  called  the  beer  game.  We  then 
describe  a  preliminary  version  of  an  instance-based,  dynamic  decision  making  model  built  using  the  ACT-R 
cognitive  architecture.  The  model  mimics  the  general  patterns  of  human  behavior  observed  for  aggregate 
performance  across  trials  and  local  performance  within  trials.  Implications  for  research  on  dynamic  decision  making 
are  summarized. 


Martinez,  S.  G.,  Talcott,  C.,  Bennett,  K.  B.,  Stansifer,  C.,  &  Shattuck,  L.  (2001).  Cognitive  Systems  Engineering 
Analyses  for  Army  Tactical  Operations.  In  In  Proceedings  of  45th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  Minneapolis,  MN. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  Cognitive  Systems  Engineering  (CSE)  approach  has  been  applied  successfully  for  domains  in  which  the 
interaction  is  primarily  driven  by  either  the  laws  of  nature  or  by  user  intentions.  We  have  initiated  a  research 
program  to  apply  the  CSE  approach  to  a  domain  that  is  driven  by  both.  We  describe  the  preliminary  results  of  our 
initial  efforts,  primarily  a  work  domain  analysis  using  the  abstraction  hierarchy  analytical  tool.  Overall,  the  CSE 
approach  has  proven  to  be  very  useful  for  analysis  and  design.  The  adversarial  nature  of  the  domain  poses  certain 
challenges.  For  example,  information  regarding  both  friendly  and  enemy  forces  is  critical  to  decision  making. 
However,  the  representations  that  have  been  developed  for  friendly  forces  cannot  be  used  to  represent  enemy  forces, 
due  to  a  lack  of  critical  information.  We  view  these  as  inherent  challenges  associated  with  this  type  of  domain, 
rather  than  as  a  shortcoming  of  the  CSE  approach. 


Matessa,  M.,  Archer,  R.,  &  Mui,  R.  (2007).  Dynamic  Spatial  Reasoning  Capability  in  a  Graphical  Interface 
Evaluation  Tool.  In  In  Proceedings  of  Eighth  International  Conference  on  Cognitive  Modeling  (ICCM  '07). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  describes  dynamic  and  spatial  reasoning  enhancements  to  the  Graph-Based  Interface  Language  tool 
(GRBIL),  which  creates  ACT-R  models  by  demonstration.  A  new  ability  for  users  to  create  monitors  enables 
procedures  to  be  dynamically  triggered.  A  new  integration  of  ACT-R  with  a  diagrammatic  reasoning  theory  allows 
ACT-R  to  perform  spatial  reasoning.  Capabilities  of  the  tool  are  demonstrated  in  a  robotic  control  task. 


Matessa,  M.,  &  Brockett,  A.  (2007).  Using  a  Diagram  Representation  System  with  ACT-R.  In  In  Proceedings  of 
Sixteenth  Conference  on  Behavior  Representation  in  Modeling  and  Simulation  (BRIMS  '07). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 
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The  ACT-R  cognitive  architecture  has  shown  an  increasing  ability  to  account  for  human  visual  information 
processing.  However,  limitations  currently  exist  that  can  be  remedied  by  the  integration  of  theories  of  diagrammatic 
representation.  A  demonstration  of  integration  of  ACT-R  with  a  diagram  representation  system  (DRS)  is  presented 
that  addresses  some  current  limitations. 


Matessa,  M.,  &  Mui,  R.  (2009).  Chapter  13.  GRBIL:  A  Tool  for  Dynamic  Operator  and  Environment  Modeling.  In 
P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and 
Technology.  Section  II.  Presenting  Battlefield  Information  to  Warfighters:  48hrBooks  by  Alion  Science  & 
Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD 
21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures 
(ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and 
Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue  SE,  Suite 
200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

Human  performance  models  can  be  very  useful  in  predicting  behavior.  However,  for  dynamic  situations  it  may  be 
difficult  to  model  both  a  changing  environment  and  the  processes  needed  to  react  to  that  environment.  This  chapter 
describes  the  Graph-Based  Interface  Language  (GRBIL)  tool  which  allows  the  modeling  of  an  environment  with  a 
task  network  model,  the  creation  of  an  interface  for  observing  that  environment  using  drag-and-drop  functionality, 
and  the  modeling  of  a  human  operator  of  that  interface  with  a  cognitive  architecture.  Using  both  modeling 
approaches  capitalizes  on  the  advantages  of  each  (the  intuitive  graphical  representation  of  task  network  modeling 
and  the  detail  of  the  cognitive  architecture).  To  enable  our  goal  of  making  the  system  usable  to  non-modeling 
experts,  operator  models  are  created  by  demonstrating  procedures  on  the  interface. 


Matthews,  M.  D.,  Strater,  L.  D.,  &  Endsley,  M.  R.  (2004).  Situation  Awareness  Requirements  for  Infantry  Platoon 
Leaders.  Military  Psychology,  16(3),  149-161. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Situation  awareness  (SA)  is  a  construct  closely  linked  to  decision  making  and  performance.  Identifying  SA 
requirements  for  specific  jobs  is  a  necessary  first  step  in  accurately  assessingSA, developing 

trainingprogramstoenhanceSAskills,andevaluating  the  impact  of  newtechnology  on  SA.  This  research  identified  the 
SA  requirements  for  infantry  small  unit  leaders.  Six  highly  experienced  infantry  subject  matter  expertswere  given 
semi-structured,  in-depth  interviews.  The  resulting  SA  requirements  hierarchy  contained  7  primary  goals  and 
multiple  subgoals.  Situation  awareness  requirements  are  job  and  mission  specific.  The  methods  used  here  may  be 
adapted  to  identifySArequirements  in  other  domains  characterized  by  small,  cohesive  teams  operating  in  dynamic 
environments  and  as  the  basis  for  developing  SA  metrics  in  these  domains. 


Mattingly,  J.,  &  Warwick,  W.  (2008).  Projectible  Predicates  in  Analogue  and  Simulated  Systems.  Synthese,  169, 
465-482. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
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Book  Chapter/Section 

Computational  modeling  ■  Analogue  modeling  ■  Human  performance  •  Theory  structure 
Summary  abstract: 

We  investigate  the  relationship  between  two  approaches  to  modeling  physical  systems.  On  the  first  approach, 
simplifying  assumptions  are  made  about  the  level  of  detail  we  choose  to  represent  in  a  computational  simulation 
with  an  eye  toward  tractability.  On  the  second  approach  simpler,  analogue  physical  systems  are  considered  that  have 
more  or  less  well-defined  connections  to  systems  of  interest  that  are  themselves  too  difficult  to  probe 
experimentally.  Our  interest  here  is  in  the  connections  between  the  artifacts  of  modeling  that  appear  in  these  two 
approaches.  We  begin  by  outlining  an  important  respect  in  which  the  two  are  essentially  dissimilar  and  then  propose 
a  method  whereby  overcoming  that  dissimilarity  by  hand  results  in  usefully  analogous  behavior.  We  claim  that 
progress  can  be  made  if  we  think  of  artifacts  as  clues  to  the  projectible  predicates  proper  to  the  models  themselves. 
Our  degree  of  control  over  the  connection  between  interesting  analogue  physical  systems  and  their  targets  arises 
from  determining  the  projectible  predicates  in  the  analogue  system  through  a  combination  of  theory  and  experiment. 
To  obtain  a  similar  degree  of  control  over  the  connection  between  large-scale,  distributed  simulations  of  complex 
systems  and  their  targets  we  must  similarly  determine  the  projectible  predicates  of  the  simulations  themselves.  In 
general  theory  will  be  too  intractable  to  be  of  use,  and  so  we  advocate  an  experimental  program  for  determining 
these  predicates. 


McCulloh,  I.,  &  Carley,  K.  M.  (2009).  Detecting  Change  in  Human  Social  Behavior  Simulation.  In  In  Proceedings 
of  Conference  at  Carnegie  Mellon  University,  School  of  Computer  Science,  Institute  for  Software  Research, 
Technical  Report  CMU-ISR-08-135. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Social  network  change  detection,  statistical  process  control,  multi-agent  simulation,  military,  organizational 
behavior,  networks,  network  statistics 

Summary  abstract: 

The  performance  of  social  network  change  detection  (SNCD)  is  evaluated  using  a  multi-agent  simulation  of 
company  level  U.S  Army  Infantry  organizations.  Agent  interaction  is  probabilistic,  with  increased  likelihood  of 
communication  based  on  similarity  in  skills,  role,  sub-unit  of  assignment,  military  rank,  and  general  personality 
homophily.  Various  social  network  measures  are  monitored  for  change  over  time  with  a  Cumulative  Sum  (CUSUM) 
control  chart,  an  Exponentially  Weighted  Moving  Average  (EWMA),  a  scan  statistic,  and  a  Hamming  Distance. 
Findings  show  that  the  average  betweenness,  the  average  closeness,  and  the  standard  deviation  of  eigenvector 
centrality  are  social  network  measures  that  are  well-suited  for  SNCD.  This  research  further  supports  the  efficacy  of 
SNCD  using  statistical  process  control  charts. 


McCulloh,  1.,  Webb,  M.,  Graham,  J.,  Carley,  K.,  &  Horn,  D.  B.  (2008).  Change  Detection  in  Social  Networks.  In  In 
Proceedings  of  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  Technical  Report  No.  ARI-TR- 
1235.  Arlington,  VA.  DTIC  ADA48461 1. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Fields  and  Groups: 

250200  -  Radio  Communications 
Descriptors: 

(U)  *  CHANGE  DETECTION,  COMMUNICATIONS  NETWORKS,  TERRORISM, 

ELECTRONIC  MAIL,  COMMAND  AND  CONTROL  SYSTEMS,  ALGORITHMS,  STATISTICAL 

PROCESSES 

Identifiers: 

(U)  *  SOCIAL  NETWORKS 
Summary  abstract: 
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Social  network  analysis  (SNA)  has  become  an  important  analytic  tool  for  analyzing  terrorist  networks,  friendly 
command  and  control  structures,  and  a  wide  variety  of  other  applications.  This  project  proposes  a  new  method  for 
detecting  change  in  social  networks  over  time,  by  applying  a  cumulative  sum  statistical  process  control  statistic  to 
normally  distributed  network  measures.  The  proposed  method  is  able  to  detect  organizational  change  in  the  same 
manner  as  a  quality  engineer  can  detect  a  change  in  a  manufacturing  process.  The  new  algorithm  is  demonstrated  on 
social  network  data  collected  on  a  group  of  24  Army  officers  going  through  a  1-year  graduate  program  at  Columbia 
University  and  on  al-Qaeda  leading  up  to  and  immediately  following  the  terrorist  attacks  of  September  11,  2001. 


McDermott,  P.,  &  Allender,  L.  (Eds.).  (2009).  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations 
and  Technology:  48hrBooks  by  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human 
Research  and  Engineering  Directorate,  http://www.arladacta.info. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book 

Summary  abstract: 
x 


McDermott,  P.,  Hutchins,  S.,  Barnes,  M.,  Koenecke,  C.,  Gillan,  D.,  &  Rothrock,  L.  (2003).  The  Presentation  of  Risk 
and  Uncertainty  in  the  Context  of  National  Missile  Defense  Simulations.  In  In  Proceedings  of  47th  Annual  Meeting 
of  the  Human  Factors  and  Ergonomics  Society  Confernece.  Denver,  CO.  (pp.  562-566). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Risk  perception  and  uncertainty  management  are  important  components  of  military  decision  making,  especially  in 
time-stressed  and  resource-limited  environments.  The  purpose  of  this  experiment  was  to  understand  the  interaction 
of  integrality  of  information,  presentation  mode,  and  information  frame  on  situation  awareness  (SA)  and  decision¬ 
making  (missile  allocation)  in  a  National  Missile  Defense  (NMD)  paradigm.  Results  of  the  information  frame 
manipulation  (expected  gain  v.  expected  loss)  support  earlier  findings  that  subjects  are  loss  averse.  SA  Accuracy 
was  higher  with  graphical  displays  than  alphanumeric  displays.  The  implications  for  NMD  are  discussed. 


McDermott,  P.  L.,  Fisher,  A.,  Allender,  L.,  &  Luck,  J.  (2008).  To  Send  or  Not  to  Send:  Does  the  Transmission  of 
Unmanned  Vehicle  Imagery  Help  or  Hinder  Soldier  Performance?  In  In  Proceedings  of  5th  Annual  Workshop  of 
Human  Factors  in  UAVs:  Human  Factors  Issues  in  Imagery  Analysis  using  Unmanned  Aerial  Vehicles,  Apache 
Junction,  AZ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

Summary 

•  Image  transmission  is  useful  to  direct  teammate  if  done  correctly 

•  Especially  useful  when  teammates  were  distributed 

•  Biggest  factor  -  does  the  recipient  know  where  he  or  she  is  in  relation  to  the  image 

•  Show  recipient  in  the  image 

•  Give  the  recipient  landmarks  so  they  can  orient  themselves 

•  UAV  imagery  provided  an  overview;  UGVs  were  better  at  detecting  enemy 
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McDermott,  P.  L.,  Barnes,  M.,  &  Hutchins,  S.  (2005).  Too  Little  or  Too  Much:  The  Effect  of  Feedback  on  Risk 
Assessment  in  a  Missile  Defense  Task.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Operators  must  often  assess  risk  and  take  actions  in  proportion  to  that  risk.  This  is  not  an  inherently  easy  task  and 
based  on  previous  experiments,  visualization  aids  may  not  be  the  entire  solution  to  aiding  operators.  This  experiment 
explored  the  impact  of  feedback  paradigms  on  performance  in  a  national  missile  defense  (NMD)  task.  Participants 
had  extensive  practice  with  one  of  three  types  of  feedback  styles:  averaged  results  after  a  set  of  scenarios,  constant 
feedback  after  every  scenario,  or  constant  feedback  with  forced  reflection.  This  study  has  three  main  contributions  to 
decision-making  research. 

1)  Although  the  trust  in  automation  literature  shows  that  averaged  feedback  increases  understanding  of  the  relative 
frequency  of  errors,  this  NMD  task  showed  that  constant  feedback  is  more  effective  than  averaged  feedback. 

2)  Participant  performance  was  still  improving  after  three  sessions,  suggesting  the  need  for  more  intense  training. 

3)  NMD  provides  a  concrete  domain  with  viable  metrics  for  investigating  the  perception  of  risk  and  uncertainty. 


McDermott,  P.  L.,  Barnes,  M.,  &  Hutchins,  S.  (2009).  *Displays  and  Decisions  -  the  Chicken  or  the  Egg?  The 
Influence  of  Displays  on  the  Decision  Making  Process.  In  In  Proceedings  of  9th  Naturalistic  Decision  Making 
Conference,  London,  England. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Decision  making  process,  design  cycle,  visualizations,  courses  of  action. 

Summary  abstract: 

Motivation  -  The  goal  is  to  encourage  discussion  on  the  philosophical  approach  to  studying  decision  making. 
Research  approach  -  A  full  design  cycle  included  cognitive  task  analysis  (CTA),  display  design,  analysis  of 
alternatives,  and  redesign.  Findings  -  The  visualizations  influenced  the  decision  process  and  the  courses  of  action. 
Research  Implications  -  This  example  illustrates  an  interesting  question  regarding  decision  making  for  envisioned 
problems  and  whether  the  designers  should  start  by  understanding  how  future  operators  are  likely  to  think  about  the 
problem  or  by  designing  alternatives  and  studying  their  impact  on  decision  making  and  course  of  action  selection. 
Originality /Value  -  Multiple  phases  of  a  naturalistic  design  process  are  discussed  as  well  as  the  impact  of  those 
displays  on  decision  strategies.  Take  away  message  -  Even  when  the  design  process  starts  with  a  CTA,  it  is 
important  to  examine  how  different  ways  of  presenting  or  interacting  with  data  effect  how  operators  think  about  the 
task  and  take  action. 


McDermott,  P.  L.,  Bolstad,  C.,  Jamieson,  G.,  Laux,  L.,  Shattuck,  L.,  &  Wickens,  C.  (2005).  Good  Design  Ideas 
Gone  Bad:  Lessons  Learned  When  System  Designs  Do  Not  Work  as  Planned.  In  In  Proceedings  of  49th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

If  at  first  you  don’t  succeed,  try  again.  Often  the  first  or  second  design  alternatives  are  modified  or  discarded  based 
on  preliminary  evaluation.  However,  there  is  much  to  be  learned  from  design  ideas  that  did  not  work  as  planned. 
Unfortunately,  the  lessons  learned  from  these  inadequate  designs  are  not  always  shared  within  the  human  factors 
community.  This  panel  provides  a  forum  to  share  the  lessons  learned  by  panelists  who  are  actually  creating  and 
evaluating  designs  in  the  field.  The  panelists  represent  a  balance  of  perspectives  from  academia,  industry,  and 
government.  Attendees  should  come  away  from  the  session  with  concrete  examples  of  inadequate  interface  designs. 
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how  they  were  improved,  and  an  understanding  of  why  the  design  did  not  work  for  a  particular  application.  A 
successful  panel  will  create  a  forum  to  share  lessons  learned  and  perhaps  prevent  practitioners  from  repeating  work 
that  has  already  been  done. 


McDermott,  P.  L.,  &  Fisher,  A.  (2009).  The  Tradeoff  of  Frame  Rate  and  Resolution  in  a  Route  Clearing  Task: 
Implications  for  Human-Robot  Interaction.  In  In  Proceedings  of  53rd  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society,  San  Antonio,  TX. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  implications  of  bandwidth  allocation  are  described  for  teleoperation  in  a  military  task  that  involved  navigation, 
target  detection,  and  target  identification.  Color  versus  grayscale  imagery  was  manipulated.  Participants  themselves 
traded  off  resolution  and  frame  rate  settings.  Participants  minimized  switching  between  resolution/frame  rate 
settings  and  tended  to  use  settings  with  high  resolution/low  frame  rate.  Courses  completed  with  the  highest 
resolution  (and  lowest  frame  rate)  had  the  fastest  target  identification  times,  but  no  other  differences  were  observed 
between  settings.  Color  imagery  offered  advantages  for  overall  course  time  and  the  time  to  identify  a  tank  as 
friendly  or  enemy. 


McDermott,  P.  L.,  Fisher,  A.,  &  Allender,  L.  (2008).  The  Transmission  of  Spatial  Route  Information  in  Distributed 
Unmanned  Vehicle  Teams.  In  In  Proceedings  of  52nd  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  22-26  September  2008.  New  York  City,  NY  USA.  New  York  City,  NY  USA:  HFES. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  purpose  was  to  understand  the  utility  of  different  methods  of  communicating  spatial  information  between 
distributed  team  members.  An  unmanned  vehicle  (UV)  operator  used  a  UV  to  scope  out  safe  routes  and  enemy 
locations  and  used  that  information  to  help  a  Soldier  conduct  a  rescue  mission.  There  were  four  communication 
methods:  visual,  verbal,  real-time  verbal  plus  visual  (VPV),  and  delayed  VPV.  We  manipulated  who  was  in 
command  to  see  the  impact  on  both  pushed  and  pulled  information.  Results  showed  advantages  of  both  visual  and 
real-time  VPV.  We  also  found  that  with  delayed  VPV,  the  Soldier  communicated  infrequently  which  resulted  in 
poorer  performance,  especially  when  the  Soldier  was  in  command.  Results  inform  how  beneficial  UV  information 
should  be  transmitted  to  Soldiers  who  need  it  in  the  field. 


McDermott,  P.  L.,  Gillan,  D.,  Riley,  J.,  &  Allender,  L.  (2009).  Chapter  16.  Five  Lessons  Learned  in  Human-Robot 
Interaction.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter: 
Foundations  and  Technology.  Section  III.  Acting  on  Battlefield  Information:  48hrBooks  by  Alion  Science  & 
Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD 
21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures 
(ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and 
Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue  SE,  Suite 
200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

The  growing  sophistication  of  robotics  systems  has  enhanced  the  utility  of  unmanned  systems  for  complex  military 
operations  (e.g.,  ordnance  disposal  or  minefield  remediation,  Lemhofer,  1999;  aerial  reconnaissance  and  patrol, 
Chappell,  2007).  The  increasing  capability  and  functionality  of  current  and  future  robotics  systems  design  requires 
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comparable  enhancements  to  robotics  interfaces  and  displays  developed  for  controlling  unmanned  systems.  Many 
military  operations  involving  robotic  assets  still  require  some  level  of  teamwork,  so  the  interfaces  must  not  only 
support  single  user  control,  but  also  promote  team  processes.  Furthermore,  displays  and  interactions  mechanisms 
must  be  designed  to  facilitate  quick  and  smooth  transitions  between  multiple  users  collaborating  to  control  multiple 
robots.  This  may  occur  under  varying  levels  of  autonomy  and  in  a  variety  of  tasks. 

Our  goal  for  this  research  was  to  improve  the  human-robot  interaction  (HRI)  through  study  of  factors  that  impact 
aspects  of  Soldier  performance  in  robotics  operations.  The  research  team  identified  key  issues  and  challenges  in 
monitoring  and  controlling  robots  with  respect  to  situation  awareness  (SA),  perception  and  performance,  and 
workload.  SA  Technologies,  Alion,  New  Mexico  State  University,  and  North  Carolina  State  University  have 
conducted  numerous  studies  to  understand  a  wide  variety  of  factors  that  will  affect  Soldiers'  interactions  with  robots. 
These  include  the  impact  of  bandwidth  reduction,  span  of  control,  unmanned  vehicle  characteristics,  level  of 
automation,  and  interface  designs  (including  control  and  feedback  from  the  robot)  as  well  as  collaborative 
technologies  and  teamwork  configurations.  The  research  has  included  both  theoretical  and  empirical  investigations 
in  a  variety  of  contexts  (e.g.,  search  and  rescue,  homeland  security  operations,  and  military  reconnaissance)  and 
experimental  mediums  (e.g.,  real-world  miniature  environments,  virtual  environments,  computer  games,  and  field 
studies).  Our  HRI  research  has  investigated  multiple  factors,  that  can  impact  SA  (and  performance)  including: 
mission  or  task  factors,  environmental  factors,  human  and  social  factors,  system  design  factors,  and  aspects  of 
interface  design.  This  paper  summarizes  five  key  lessons  learned  from  this  multiple-  year  research  program  within 
the  Advanced  Decision  Architectures  Collaborative  Technology  Alliance: 

1.  System  and  interface  design  must  support  the  multiple  facets  of  SA. 

2.  Operating  a  robot  inherently  requires  multitasking.  One  solution  to  the  human  performance  costs  of  multitasking 
is  to  automate  certain  tasks,  but  the  effects  of  automation  on  performance  may  be  limited  by  human  and  social 
factors. 

3.  Soldier-robot  interfaces  need  to  support  effective  teamwork  and  the  communication  of  spatial  information. 

4.  Soldiers  need  interface  interventions  and  compensatory  methods  to  ameliorate  the  effects  of  remote  presence 
(e.g.,  reduction  on  natural  ability  to  perceive,  move,  and  manipulate  objects  in  an  environment)  that  can  hinder 
navigation  and  limit  Soldier  ability  to  project  his  intent  upon  the  environment. 

5.  Performance  measures  must  be  examined  in  tandem  with  SA  measures  to  get  better  assess  the  effects  of  HRI 
factors  on  overall  system  performance. 


McDermott,  P.  L.,  &  Hautamaki,  B.  (2007).  Resource  Allocation  Decisions:  The  Impact  of  Individual  Differences 
on  Preferences  and  Performance.  In  In  Proceedings  of  K.Mosier  &  U.  Fischer  (eds.),  Proceedings  of  the  Eighth 
International  Conference  on  Naturalistic  Decision  Making. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

Key  findings: 

A  “golden”  criteria  can  be  more  important  than  others,  but  the  specific  criteria  changes  depending  on  the  situation 
demands. 

The  info  on  the  decision  aid  is  just  part  of  what  makes  up  the  decision;  decisions  are  made  in  the  context  of  METT- 
TC. 

An  understanding  of  the  specific  info  considered. 

Current  systems  solve  for  the  optimal  location  of  one  sensor;  they  do  not  address  the  configuration  of  a  suite  of 
sensors. 


McDermott,  P.  L.,  Luck,  J.,  Allender,  L.,  &  Fisher,  A.  (2005).  Effective  Human  to  Human  Communication  of 
Information  Provided  by  an  Unmanned  Vehicle  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors 
and  Ergonomics  Society.  Orlando,  FL. 
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Summary  abstract: 

Much  of  the  research  on  unmanned-vehicles  (UVs)  focuses  on  technology  or  interface  design.  This  study  however, 
investigated  how  to  best  support  effective  communication  between  the  operator  monitoring  a  UV  and  the  Soldier  in 
the  field  using  that  information  to  complete  a  mission.  Several  questions  arise:  Does  the  operator  need  to  be  co¬ 
located  with  Soldiers  in  the  field  or  can  he  or  she  be  in  a  more  secure  rearward  location?  Does  the  team  need  the 
capability  to  transmit  visual  images  or  is  radio  communication  adequate?  Is  information  from  one  type  of  UV  better 
than  others?  Do  real  time  mapping  and  tracking  technologies  increase  situation  awareness  (SA)?  To  begin  to  answer 
these  questions,  military  teams  conducted  rescue  missions  using  the  video  game  Raven  Shield  as  a  simulated 
battlefield.  The  analysis  of  performance  data,  self  reports,  and  observations  provide  some  valuable  insight  to  these 
questions. 


McNeese,  M.  D.,  Bains,  P.,  Brewer,  I.,  Brown,  C.,  Connors,  E.  S.,  Tyrone  Jefferson,  J.,  Jones,  R.  E.  T.,  &  Terrell,  I. 
(2005).  The  Neocities  Simulation:  Understanding  the  Design  and  Experimental  Methodology  Used  to  Develop  a 
Team  Emergency  Management  Simulation  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  describes  the  continued  development  of  a  scaled-world  simulation  designed  to  conduct  empirical 
research  on  team  cognition  and  decision-making  within  a  distributed  environment.  The  NeoCITIES  simulation  was 
created  to  study  decision-making  and  the  impact  of  hidden  knowledge  profiles  on  team  performance  within  a 
distributed  command,  control,  and  communications  (C3)  setting.  NeoCITIES  has  been  designed  for  the  purpose  of 
representing  both  new  and  operationally  relevant  scaled  worlds,  while  emulating  the  complexities  and  attributes  of 
emergent  decision-making  scenarios  involving  emergent  counterterrorism  events.  Because  patterns  of  activity 
emerge  across  time,  knowledge  is  often  hidden  and  disconnected  within  and  across  teams.  NeoCIITES  has  been 
orchestrated  to  assess  and  evaluate  the  extent  to  which  teams  can  socially  construct  knowledge  while  ineteracting 
through  various  means  of  technological  support.  Specifically,  NeoCITIES  is  an  interactive  computer  program 
designed  to  display  information  pertaining  to  events  and  occurrences  in  a  virtual  city  space.  The  teams  in  the 
simulation  represent  three  separate  services  (e.g.,  Police,  Fire/EMS,  and  Hazmat)  in  which  they  must  assess 
situations,  interact  and  communicate  according  to  their  inter-team  and  intra-team  roles,  allocate  resources  in  a  timely 
manner,  and  make  decisions  within  the  context  of  emergency  crisis  management.  Once  NeoCITIES  development 
has  been  completed,  the  simulation  will  be  used  as  an  experimental  task  to  measure  the  impact  of  hidden  knowledge 
profiles  on  teamwork  and  decision-making  in  the  distributed  team  context. 


Militello,  L.  G.,  &  Hoffman,  R.  R.  (2008).  The  Forgotten  History  of  Cognitive  Task  Analysis.  In  In  Proceedings  of 
52nd  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  22-26  September  2008.  New  York  City,  NY 
USA.  New  York  City,  NY  USA:  HFES. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Many  reports  of  recent  research  on  topics  in  cognitive  systems  engineering  describe  their  methods  by 
distinguishing  cognitive  task  analysis  from  — traditional  or  — behavioral  task  analysis.  A  hallmark  of  modern 
cognitive  task  analysis  (CTA)  methods  is  that  they  place  primary  focus  on  understanding  the  cognitive  demands  of  a 
task  and  the  knowledge  and  strategies  that  underlie  performance.  While  cognitive  task  analysis  may  have  seemed  to 
be  a  revolutionary  approach  introduced  in  recent  years,  a  review  of  the  history  of  task  analysis  reveals  many  things 
that  have  been  lost  in  modern  treatments  of  the  history  of  human  factors.  This  presentation  will  review  some 
highlights  of  this  forgotten  history,  including  the  ideas  and  methods  of  the  Psychotechnicians,  the  earliest  industrial 
psychologists,  the  Taylorists,  and  others  who  contributed  to  modern  CTA.  Task  analysis  never  lacked  cognitive 
categories.  Even  microscale  time  and  motion  studies  involved  the  analysis  of  the  work  of  domain  experts.  Basic 
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ideas  of  human-machine  systems  and  of  complexity  also  appear  in  some  of  the  earliest  literatures  industrial 
psychology  of  the  first  decades  of  the  20th  century. 


Militello,  L.  G.,  Patterson,  E.  S.,  Bowman,  L.,  &  Wears,  R.  (2007).  Information  Flow  During  Crisis  Management: 
Challenges  to  Coordination  in  the  Emergency  Operations  Center.  Cognition,  Technology  and  Work,  9(1),  25-31. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Emergency  operations  center  Crisis 
management  Team  decision  making  Team 
coordination  Cross  agency  collaboration 

Summary  abstract: 

This  paper  discusses  challenges  to  coordination  in  the  emergency  operations  center  (EOC).  A  county-level  EOC  is 
made  up  of  representatives  from  a  range  of  organizations  including  local  government,  fire,  police,  hospital,  utility, 
and  Red  Cross  representatives.  These  ad  hoc  teams  are  tasked  with  coming  together  during  an  emergency  to  obtain 
and  deliver  resources  to  first  responders  on  the  scene  of  the  disaster,  as  well  coordinating  transportation  of 
casualties,  tracking  of  fatalities,  and  establishment  of  shelters.  Two  county-level  exercises  were  observed.  Themes 
were  identified  across  the  two  exercises.  Recommendations  for  better  supporting  coordination  in  crisis  management 
are  offered. 


Miller,  J.  E.,  Patterson,  E.  S.,  &  Woods,  D.  D.  (2003).  Elicitation  by  Critiquing:  An  Exploratory  Study.  In  In 
Proceedings  of  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Knowledge  elicitation  methods  uncover  information  about  how  a  practitioner  works  in  a  field.  However,  these 
methods  have  the  challenges  of  grounding  in  context,  accessibility  to  experts  and  tasks,  being  laborious  and  lime 
consuming,  and  repeatability.  This  study  investigated  a  critiquing  methodology’s  ability  to  address  these  challenges 
while  still  being  authentic  and  generative.  The  domain  investigated  was  military  intelligence  analysis,  a  field  that 
experiences  a  constant  turnover  of  workers.  To  maintain  the  highest  level  of  productivity,  the  expertise  of  their 
military  intelligence  analysis  practitioners  must  be  understood.  A  study  was  done  involving  a  novice  performing  a 
basic  analysis  task.  Then,  six  experts  with  various  backgrounds  critiqued  the  novice’s  process.  The  results  suggest 
that  the  critiquing  method  addresses  the  challenges  of  knowledge  elicitation  methods  while  being  authentic  and 
generative.  The  specific  guiding  question  for  addressing  both  issues  was  ‘Can  the  critiquing  method  be  used  to  help 
unveil  expertise  in  intelligence  analysis?’ 


Miller,  N.  L.,  &  Shattuck,  L.  G.  (2004).  A  Process  Model  of  Situated  Cognition  in  Military  Command  and  Control. 
In  In  Proceedings  of  Command  and  Control  Research  and  Technology  Symposium  (9th),  held  in  San  Diego,  CA, 
Jun  2004.  Published  in  the  Proceedings  of  the  Command  and  Control  Research  and  Technology  Symposium  (9th), 
2004:  The  Power  of  Information  Age  Concepts  and  Technologies.  San  Diego,  CA.  DTIC  ADA466093. 
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(U)  *COGNlTION,  *COMMAND  AND  CONTROL  SYSTEMS,  METHODOLOGY,  HUMAN 
FACTORS  ENGINEERING,  MAN  MACHINE  SYSTEMS,  CASE  STUDIES,  SYMPOSIA 

Summary  abstract: 

Complex  cognitive  systems  couple  humans  with  machines  for  the  purpose  of  accomplishing  a  specific  goal.  It  is 
often  the  case  that  human  factors  practitioners  focus  their  attention  on  the  humans  while  designers  tend  to  focus  on 
the  technological  aspects  of  the  system.  The  point  of  intersection  between  humans  and  technology  has  become  a 
boundary  with  respect  to  evaluation.  In  addition,  human  factors  practitioners  have  often  studied  the  result  of 
cognitive  activity  (e.g.,  a  decision)  rather  than  the  processes  that  lead  to  the  outcome.  In  this  paper,  the  authors 
present  a  general  model  that  combines  the  technological  aspects  of  a  system  with  the  perceptual  and  cognitive 
processes  of  the  humans  embedded  in  the  system.  The  model  emphasizes  that  such  systems  are  both  process 
oriented  and  dynamic.  The  authors  describe  a  process  tracing  methodology  that  can  be  used  to  investigate  the  flow 
of  data  and  information  through  both  the  technological  and  human  components  of  the  system.  The  attack  on  the  USS 
Stark  is  used  as  a  case  study  to  illustrate  the  model  and  the  process  tracing  methodology.  The  results  of  the  process 
tracing  analysis  have  implications  for  the  design  of  complex  systems  and  the  training  received  by  those  who  operate 
such  systems. 


Miller,  T.  E.,  Stanard,  T.  W.,  &  Wiggins,  S.  L.  (2003).  Applying  Decision  Centered  Design  to  Damage  Control  for 
the  Navy.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society  Conference. 
Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Decision-Centered  Design  (DCD)  was  created  so  that  the  users  of  information  technology  (IT)  have  an  advocate  in 
the  design  cycle.  DCD  was  created  to  increase  the  value  of  IT  for  the  decision  maker.  It  is  a  cognitive  engineering 
approach  to  design  that  uncovers  and  models  the  cognitive  expertise  of  users  and  conveys  these  insights  to  IT 
developers.  DCD  ensures  that  cognitive  challenges  are  addressed  in  IT  designs  so  that  cognitive  performance  is 
improved.  Other  cognitive  engineering  approaches  share  these  same  goals,  but  DCD  also  has  principles  and 
processes  that  other  cognitive  engineering  approaches  may  not  share.  This  paper  will  use  a  Damage  Control  decision 
support  system  to  illustrate  the  application  of  DCD. 


Montanari,  R.,  Toninelli,  A.  and  Bradshaw,  J.  M.  (2005).  Context-Based  Security  Management  for  Multi-Agent 
Systems.  In  In  Proceedings  of  Second  IEEE  Symposium  on  Multi-Agent  Security  and  Survivability  (MASS  2005), 
Philadelphia,  PA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Policies  are  being  increasingly  used  for  controlling  the  behavior  of  complex  multi-agent  systems.  The  use  of  policies 
allows  administrators  to  specify  both  agent  permissions  and  duties  without  changing  source  code  or  requiring  the 
consent  or  cooperation  of  the  agents  being  governed.  However,  policy-based  control  can  encounter  difficulties  when 
applied  to  agents  that  act  in  pervasive  environments  characterized  by  frequent  and  unpredictable  changes.  In  this 
case,  policies  cannot  be  all  specified  a  priori  to  face  any  operative  run  time  situation,  but  require  continuous 
adjustments  to  allow  agents  to  behave  in  a  contextually  appropriate  manner.  Current  approaches  to  policy 
representation  have  been  restrictive  in  many  ways,  as  they  typically  follow  a  subject-centric  model,  which  assigns 
agent  permissions  and  obligations  on  the  basis  of  agent  role/identity  information.  However,  in  the  new  pervasive 
scenario  the  roles/identities  of  interacting  agents  may  not  be  known  a-priori  and  most  important,  may  not  be 
informative  or  sufficiently  trustworthy.  We  claim  that  the  design  of  policy -based  agent  systems  for  pervasive 
environments  requires  a  paradigm  shift  from  subject-centric  to  contextcentric  policy  models.  This  paper  discusses 
some  issues  concerning  the  specification  and  enforcement  of  contextdriven  policies  and  presents  a  novel  context- 
based  policy  approach  that  considers  context  as  a  first-class  principle  to  guide  both  policy  specification  and 
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enforcement.  In  this  perspective,  “context”  explicitly  appears  in  the  specification  of  security  policies  and  context 
changes  trigger  the  evaluation  process  of  applicable  agent  permissions  and  obligations. 


Moon,  B.  M.  (2002).  Naturalistic  Decision  Making:  Establishing  a  Naturalistic  Perspective  in  Judgment  and 
Decision  Making  Research.  In  In  Proceedings  of  19th  Qualitative  Analysis  Conference,  May  23-25,  2002, 
McMaster  University,  Hamilton,  Ontario,  Canada. 


Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
NDM;  Naturalistic  Decision  Making 

Summary  abstract: 

Within  the  psychological  sciences,  and  the  specialized  field  of  judgment  and  decision  making  in  particular,  a 
growing  number  of  international  researchers  have  coalesced  around  the  concept  of  a  movement  called  “naturalistic 
decision  making  (NDM).”  While  strongly  sensing  that  their  work  stands  apart  from  “laboratory”  or  “traditional 
decision  research,”  they  have  struggled  somewhat  to  articulate  a  shared  boundary  between  their  movement  and  other 
researchers  within  their  field.  Various  attempts  to  distinguish  NDM  have  focused  on  different  characteristics  of  the 
subject  of  study,  aspects  of  the  situation  of  study,  and  the  study  setting  itself.  None,  however,  have  sought  to  draw 
boundaries  on  the  basis  of  what  it  means  to  be  a  naturalist;  that  is,  on  the  perspective  a  naturalist  takes  and  the 
methodological  position  a  naturalist  stands  in.  This  paper  offers  an  overview  of  the  NDM  movement,  highlights  the 
attempts  to  establish  its  boundaries  and  critical  remarks  of  those  attempts,  then  discusses  the  naturalist  perspective 
and  methodological  position.  A  suggestion  is  made  that  the  adoption  of  the  naturalistic  perspective  can  serve  as  the 
common  ground  for  the  NDM  community  and  deal  effectively  with  the  common  criticisms  waged  against  it. 


Moon,  B.  M„  Hoffman,  R.  R.,  Shattuck,  L.  G.,  Coffey,  J.  W.,  Goodman,  P.,  Linn,  R.,  Lang,  J.,  &  Sullivan,  M. 
(2008).  Rapid  and  Accurate  Idea  Transfer:  Evaluating  Concept  Maps  against  Other  Formats  for  the  Transfer  of 
Complex  Information.  In  A.  J.  Canas,  P.  Reiska,  M.  Ahlberg  &  J.  D.  Novak(  Eds)  Proceedings  of  the  Third 
International  Conference  on  Concept  Mapping,  Concept  Mapping:  Connecting  Educators.  Tallinn  University, 
Estonia  &  Helsinki,  Finland  2008. 
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Summary  abstract: 

Concept  Maps  (Cmaps)  have  proven  useful  for  capturing  and  organizing  knowledge,  particularly  of  those  who 
create  the  Cmaps.  Their  usefulness  for  transferring  knowledge  has  not  been  as  extensively  researched.  In  business, 
government  and  military  settings,  slideshows  (e.g.,  Microsoft  PowerPoint)  are  a  preferred  format  for  presenting 
“complex”  ideas,  while  the  journal  article  remains  the  preferred  format  in  academic  circles.  This  empirical 
investigation  compared  Cmaps  with  other  common  formats  of  information  presentation.  Complex  information 
expressed  in  a  report  about  a  humanitarian  crisis  was  presented  to  military  graduate  students  in  four  formats:  journal 
article  in  traditional  text,  a  hypertext  version  of  the  journal  article,  a  Microsoft  PowerPoint  presentation,  and  a  set  of 
hyperlinked  and  resourced  Cmaps.  After  reviewing  the  material,  participants  recreated  the  information  and  answered 
a  set  of  questions  to  test  comprehension  of  the  material.  Additionally,  a  preference  survey  was  conducted  with  34 
professionals  across  military  and  industry  domains  to  consider  attitudes  toward  Cmaps  for  presentation.  Cmaps  were 
empirically  demonstrated  to  be  more  effective  than  PowerPoint  on  key  measures  of  knowledge  transfer  and  rapidity 
in  creation.  They  were  also  shown  to  be  the  preferred  format  for  complex  information  presentation,  when  compared 
to  PowerPoint. 
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Moon,  B.  M.,  Wei,  S.,  &  Cox,  D.  A.  (2004).  Cognitive  Impact  Metrics:  Applying  Macrocognition  During  the 
Design  of  Complex  Cognitive  Systems.  In  In  Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  New  Orleans,  LA.  (pp.  473-477). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Macrocognition  is  an  emerging  theoretical  and  methodological  framework  for  describing  cognitive  work  as  it 
naturally  occurs  (Klein,  Ross  et  al.,  2003).  It  can  form  the  basis  for  the  design  of  complex  cognitive  systems  that 
augment,  rather  than  degrade,  proficient  performance.  This  paper  presents  a  method  for  using  macrocognition  during 
design  to  anticipate  how  a  complex  cognitive  system  will  impact  cognition.  We  have  developed  a  suite  of  metrics 
we  call  “Cognitive  Impact  Metrics”  (CIM).  Because  they  highlight  the  necessary  features  of  and  potential  barriers  to 
proficient  cognitive  performance,  these  metrics  and  their  associated  measures  provide  us  with  a  framework  in  which 
we  can  generate  predictions  about  where  and  how  our  system  will  enhance  or  hinder  our  performance.  Application 
of  CIM  may  be  particularly  useful  in  the  design  of  systems  where  many  potential  applications  must  be  culled  down 
to  a  more  manageable  set  of  candidates. 


Moon,  I.,  &  Carley,  K.  M.  (2007).  Modeling  and  Simulation  of  Terrorist  Networks  in  Social  and  Geospatial 
Dimensions.  IEEE  Intelligent  Systems:  Special  issue  on  Special  issue  on  Social  Computing. 
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Summary  abstract: 

A  simple  theoretical  multiagent  model  reasons  about  the  criticality  of  terrorists  and  regions  as  terrorist  interactions 
coevolve  in  geographical  and  social  spaces. 

Where  we  are  influences  who  we  know,  and  vice  versa.  As  we  move  to  new  cities  or  countries,  our  contacts  change. 
For  instance,  when  a  company  relocates  its  employees,  they  develop  new  working  relations  with  others  while  they 
perform  assigned  tasks.  In  theory,  these  relocations  should  improve  company  performance.  1  However,  performance 
also  depends  on  individuals  knowing  who  to  ask  about  what — that  is,  on  transactive  memory. 2  Moving  disrupts 
transactive  memory  and  the  social  relations  by  which  information  flows.  So,  the  question  arises  whether 
performance  can  improve  when  social  and  geospatial  distributions  change  simultaneously.  Social  and  spatial 
relations  evolve  over  time.  Estimating  their  evolutions  is  important  for  management,  command  and  control 
structures,  and  intelligence  analysis  research.  By  knowing  future  agent  social  and  spatial  distributions,  an  analyst 
can  identify  emergent  leaders,  hot  spots,  and  organizational  vulnerabilities.  Historically,  such  estimations  have 
depended  heavily  on  qualitative  data  analyses  by  subject-matter  experts. 3  A  few  researchers  approached  the  issue 
using  multiagent  models  and  simulation.  The  models  addressed  the  complex  nature  of  the  organization  and  task 
assignments,  resource  distributions,  or  agent  locations.  The  simulations  addressed  the  near-term  organizational 
changes.  This  research  came  from  two  perspectives:  the  effects  of  change  in  the  social  network4,5  and  the  effects  of 
geospatial  change. 6, 7  Both  perspectives  can  project  aspects  of  emerging  organizational  structure  and  future 
performance,  but  they  can’t  examine  the  interaction  between  physical  and  social  movements.  We’ve  developed  a 
simple  theoretical  multiagent  simulation  model  to  show  how  changes  in  the  coevolution  of  social  and  geospatial 
dimensions  affect  group  behavior.  Our  model  overcomes  the  limitations  of  isolated  social  and  spatial  models  (see 
the  “Related  Work  in  Social  and  Geospatial  Modeling”  sidebar).  To  illustrate  the  model’s  potential  for  reasoning, 
we  examine  its  implications  here  for  a  real-world  terrorist  network,  using  data  extracted  from  open  source  texts. 
Although  a  full  validation  would  require  additional  field  data,  the  model’s  output  reveals  important  aspects  of 
complex  organizational  evolution  that  apply  beyond  the  counterterrorism  domain. 
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Moon,  I.-C.,  &  Carley,  K.  M.  (2007).  Evolving  Multi-Agent  Network  Structure  with  Organizational  Learning.  In  In 
Proceedings  of  SpringSim'07,  Norfolk,  VA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Discrete  event  simulation,  agent  based  social  simulation,  organizational  learning,  organization  structure 

Summary  abstract: 

Organizational  structure  changes  over  time  due  to  various  reasons,  such  as  organizational  learning,  situation  changes 
and  personnel  turnovers,  etc.  Estimating  the  structure  changes  will  reflect  organizational  performance  changes, 
emergent  leaders,  new  key  links,  etc.  This  paper  introduces  a  multi-agent  model  that  simulates  the  organizational 
structure  evolution  over  time.  The  simulated  structure  evolution  will  be  driven  by  the  organizational  learning 
procedure  that  we  devised.  We  perform  virtual  experiments  with  two  distinct  cases,  an  organization  with  the 
learning  mechanism  and  the  other  one  without  learning.  The  performances  of  the  two  case  organizations  were 
examines  under  situation  change  assumptions.  The  organization  with  learning  mechanism  was  better  than  the  other 
when  situation  changes  were  predictable.  We  also  scan  the  network  topology  changes  over  time,  and  we  identified 
that  the  average  distance  among  the  nodes  gets  smaller  as  learning  proceeds.  This  work  is  a  preliminary  effort  to 
examine  the  effect  of  organizational  learning  and  to  formulate  the  evolution  of  organizational  structures. 


Moon,  I.-C.,  &  Carley,  K.  M.  (2007).  Locating  Optimal  Destabilization  Strategies.  In  In  Proceedings  of  ICCRTS. 
San  Diego,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Network  destabilization  is  a  critical  tactic  for  disrupting  organizations  such  as  terrorist  networks  or  organized  crime 
networks.  For  any  network,  there  are  many  possible  tactics  that  could  be  used  to  destabilize  it.  For  example,  a  set  of 
nodes  (people)  could  be  removed  or  added.  So  could  the  links  among  the  nodes.  Even  if  we  limit  ourselves  to  node 
removal,  there  are  issues  which  nodes  to  remove  and  when.  Thus,  an  automated  framework  for  the  generation  of  the 
destabilization  scenarios  is  useful,  for  example,  in  what-if  scenario  analysis  and  vulnerability  analysis.  We  created 
1024  randomly  generated  strategies  and  1024  learning  algorithm-based  strategies  for  the  destabilization  of  a  sample 
terrorist  network.  The  learning  algorithm  based  strategies  were  more  effective  in  reducing  the  performance  of  the 
network.  Moreover,  we  analyzed  the  preference  of  the  learning  algorithm  and  discovered  efficient  tactics  such  as 
undermining  the  network  first  and  isolating  the  knowledge  sources  of  a  network  later.  We  believe  that  the  presented 
method  can  be  applied  to  other  disciplines  requiring  a  simulation  and  a  what-if  scenario  generation  involving 
network  destabilization. 


Moon,  I.-C.,  &  Carley,  K.  M.  (2007).  Self-Organizing  Social  and  Spatial  Networks  under  What-if  Scenarios.  In  In 
Proceedings  of  AAMAS  2007.  Honolulu,  Hawaii. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Categories  and  Subject  Descriptors 

[Distributed  Artificial  Intelligence]:  Multi- Agent  Systems 

General  T erms 

Algorithms,  Experimentation 

Keywords 

Network  evolution,  Multi-agent  system,  Organizational  structure 
Summary  abstract: 

Multi-agent  models  have  been  used  to  simulate  complex  systems  in  many  domains,  In  some  models,  the  agents 
move  in  a  physical/grid  space  and  are  constrained  by  their  locations  on  the  spatial  space,  e,g,  Sugarscape,  In  others, 
the  agents  interact  in  a  social  multi-dimensional  space  and  are  bound  to  their  knowledge  and  social  positions,  e,g, 
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Construct.  However,  many  real  world  problems  require  a  mixed  model  containing  both  spatial  and  social  features, 
This  paper  introduces  such  a  multi  agent  system,  Construct-Spatial,  which  simulates  agent  communication  and 
movement  simultaneously.  It  is  an  extended  version  of  Construct,  which  is  a  multi-agent  social  model,  and  its 
extension  is  based  on  a  multi-agent  grid  model,  Sugarscape,  To  understand  the  impact  of  this  integration  of  the  two 
spaces,  we  run  virtual  experiments  and  compare  the  output  from  the  combined  space  to  those  from  each  of  the  two 
spaces,  The  initial  analysis  reveals  that  the  integration  facilitates  unbalanced  knowledge  distribution  across  the 
agents  compared  to  the  grid-only  model  and  limits  agent  network  connections  compared  to  the  social  network  model 
without  spatial  constraints,  After  the  comparisons,  we  setup  what-if  scenarios  where  we  varied  the  type  of  the  threats 
faced  by  network  and  observe  their  emergent  behaviors,  From  the  what-if  analyses,  we  locate  the  best 
destabilization  scenario  and  find  the  propagation  of  the  effects  from  the  spatial  space  to  the  social  network  space. 

We  believe  that  this  model  can  be  a  conceptual  model  for  assessing  the  efficiency  and  the  robustness  of  team 
deployments,  network  node  distributions,  sensor  distributions,  etc. 


Moreau,  L.,  Bradshaw,  J.,  Breedy,  M.,  Bunch,  L.,  Hayes,  P.,  Johnson,  M.,  Kulkarni,  S.,  Lott,  J.,  Suri,  N.,  &  Uszok, 
A.  (2005).  Behavioural  Specification  of  Grid  Services  with  the  KAoS  Policy  Language.  In  In  Proceedings  of  Cluster 
Computing  and  Grid  2005  (CCGrid05),  Cardiff,  UK. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Complex  services  in  Service-Oriented  Architectures  such  as  the  Grid  typically  require  to  be  configured  in  multiple 
ways  that  cannot  be  anticipated  by  service  designers;  we  illustrate  this  requirement  by  studying  the  myGrid  registry, 
a  Grid  Registry  capable  of  supporting  annotations  of  service  descriptions  by  third-party  users.  Instead,  services  have 
to  be  conceived  so  that  they  can  be  configured  at  deployment  and  run  time.  We  argue  that  KAoS  is  a  powerful 
andflexible  language  that  can  help  define  such  configurations.  Using  our  registry  case  study,  we  examine  the 
requirements  that  the  definition  of  such  complex  configurations  brings  on  policy  languages  and  explain  how  they 
can  be  satisfied.  Specifically,  we  use  role-value  maps  to  express  constraints  betweenproperty  values;  we  introduce  a 
notion  of  Policy  Set  with  associated  parameters  that  support  constraints  within  a  well  defined  scope;  finally,  we 
define  a  notion  of  Context  that  allows  us  to  refer  to  property  values  that  were  extant  in  past  execution  environments. 
Essentially,  these  concepts  allow  us  to  add  constraints  to  values  in  policy  definitions,  to  organise  policies  in  coherent 
and  structure  blocks,  and  to  refer  to  the  execution  history.  The  paper  discusses  these  concepts  and  how  they  are 
implemented  in  a  binding  ofthe  KAoS  policy  language  to  the  myGrid  Registry. 


Morison,  A.,  Voshell,  M.,  Roseler,  A.,  Feil,  M.,  Tittle,  J.,  Tinapple,  D.,  &  Woods,  D.  D.  (2009).  Chapter  10. 
Integrating  Diverse  Feeds  to  Extend  Human  Perception  into  Distant  Scenes.  In  P.  McDermott  and  L.  Allender 
(Eds.),  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  II.  Presenting 
Battlefield  Information  to  Warfighters:  48hrBooks  by  Alion  Science  &  Technology  Corporation  under  contract  to 
the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and  Engineering 
Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology 
Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision 
Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr- 
webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

New  technological  capabilities  create  opportunities  but  also  new  challenges  as  people  attempt  to  carry  out  even 
more  demanding  tasks  in  conflict  and  crisis  situations.  This  chapter  examines  the  impact  of  advances  in  sensing, 
networking,  and  robotic  technologies.  These  capabilities  allow  people  in  remote  locations  to  apprehend  and  explore 
aspects  of  the  world  as  if  they  were  standing  in  places  that  are  too  difficult,  dangerous,  or  impossible  to  occupy. 
Investments  in  these  technologies  are  predicated  often  on  the  belief  that  they  will  produce  increased  situation 


227 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 

awareness  for  the  remote  human  observer.  However,  as  prototypes  are  rolled  out,  automation  surprises  occur  which 
reveal  the  risk  of  unanticipated  and  unintended  breakdowns  in  awareness. 

For  example,  urban  search  and  rescue  workers  using  robots  to  search  inaccessible  areas  of  debris  found  that 
interpreting  the  video  feeds  was  a  slow  deliberative  process  instead  of  automatic  reliable  perceptual  experience.  It 
was  difficult  to  accurately  apprehend  three-dimensional  surface  layout,  the  relative  size/mass  of  objects,  the  robot's 
ability  to  traverse  obstacles,  the  robot's  speed,  and  the  robot's  local  orientation.  The  robots  acted  as  a  stand-in  for  a 
remote  human  observer,  mediating  the  observer's  ability  to  perceive  and  act  in  the  scene  of  interest  (in  this  case,  the 
feed  was  a  video  image  from  a  single,  forward  looking  camera).  As  a  result,  the  remote  observer  is  partially 
decoupled  from  many  of  the  information-rich  properties  of  the  scene  that  would  be  available  if  the  observer  were 
present  in  that  scene.  Similar  findings  have  come  from  other  recent  cases  of  using  robots  in  search  tasks  where 
remote  observers  experience  serious  difficulties  to  understand  the  distant  scene  being  traversed  by  a  robotic  system. 
Based  on  such  findings,  we  began  a  line  of  research  to  innovate  new  ways  to  bring  distant  feeds  from  robots  and 
sensor  nets  into  the  range  of  human  perceptual  skills.  The  work  began  by  exploring  ways  to  "break  the  tyranny  of 
the  flat  framed  screen"  as  the  default  display  medium  in  order  to  overcome  the  keyhole  effect.  In  exploring  different 
possibilities,  we  were  developing  possible  uses  of  3D  (three  dimensional)  visualizations.  In  this  work  we  found  that 
3D  technology  actually  represents  control  of  point  of  view  and  shifts  in  perspective.  This  is  in  contrast  to  the  usual 
approach  which  tries  to  compare  the  impact  of  2D  versus  3D  displays. 

Other  studies  began  to  examine  how  the  new  sensing  technology  influenced  coordination  over  multiple  groups 
(connecting  multiple  perspectives)  and  commander  decision  making.  In  one  study  incident  commanders  viewed  a 
video  feed  from  a  UAV  as  they  responded,  from  a  distance,  to  a  simulated  cascading  chemical  plant  accident.  The 
results  showed  that  incident  commanders  over-relied  on  the  video  feed  undemtilizing  other  sources  of  information 
and  missed  important  events  embedded  in  the  scenario. 

Emerging  form  this  work  was  the  realization  that  the  key  leverage  point  to  achieve  high  performance  human- 
technology  systems  was  to  explicitly  coordinate  multiple  perspectives  across  multiple  roles  and  scales.  Multiple 
potential  design  solutions  were  developed  based  on  this  insight  about  how  to  extend  perception-including  concepts 
for 

•  perspective  folding  for  feeds  from  UGVs. 

•  adaptive  focus  based  on  event  recognition  and  spatial/temporal  center-surround 

•  perspective  control  (Roesler,  2005;  Morison,  in  preparation)  See 
http://cseLeng.ohiostate.edu/productions/diversefeeds  for  demonstrations  of  the  concepts. 

In  terms  of  human-robot  interaction,  extending  perception  concepts  led  to  performance  advantages  arising  from 
coordinating  shared  perspectives  resulted  in  more  efficient  navigation,  search,  and  rescue  in  a  staged-world 
emergency  response  simulation.  At  a  larger  system  scale,  the  challenge  of  synchronizing  multiple  perspectives  over 
multiple  roles  and  echelons  in  netcentric  operations  was  examined  using  disaster  response  as  an  empirical  test  case. 
The  chapter  provides  a  roughly  historical  account  ofthe  research  on  the  theme  of  extended  perception  conducted 
within  the  context  of  the  ARL  ADA  collaborations.  It  begins  with  reframing  the  distinction  between  two- 
dimensional  and  three-dimensional  displays  based  on  past  findings  on  human  perception.  These  findings  illustrate 
that  in  a  three-dimensional  environment  the  fundamental  opportunity  afforded  by  the  new  technology  is  control  of 
point-of-view. 

With  this  insight  in  mind,  the  next  section  presents  new  design  concepts  to  widen/overcome  the  keyhole  typically 
found  in  HRI  displays.  Each  of  these  designs  leverage  a  three-dimensional  virtual  environment  to  coordinate 
perspective  taking  and  even  support  multiple  simultaneous  perspectives.  In  assessing  the  advantage  of  the  new 
display  concepts  in  a  navigation,  search,  and  identify  task,  a  novel  metric  for  quantifying  path  complexity  is 
introduced. 

A  new  display  concept  for  adaptive  focus  is  presented  within  a  surveillance  context.  This  design  overcomes  the 
keyhole  of  a  single  pan-tilt-zoom  (PTZ)  video  camera  with  a  narrow  field-of-view  (FOV).  Another  display  concept 
expands  control  of  point-of-view  to  enable  multiple  people  to  take  advantage  of  a  single  feed  from  single  robot.  The 
design  coordinates  the  perspectives  of  a  robot  handler,  a  rescue  worker,  and  a  robot  in  a  search  and  rescue  task. 
Included  with  this  theme  of  coordinating  shared  perspectives,  is  the  need  to  include  synchronizing  activity  within 
netcentric  operations.  These  netcentric  operations  include  checkpoint  operations  in  a  military  setting,  urban  fire 
fighting,  and  managing  a  convoy  of  walking  wounded. 

The  final  section  introduces  a  theory  of  perspective-taking  in  virtual  and  physical  environments.  Perspective -taking 
is  the  process  of  changing  the  point  of  observation  relative  to  a  scene  in  order  to  understand  the  properties,  objects, 
actors,  and  activities  that  are  of  interest  to  the  observer.  Supporting  perspective-taking  effectively  is  a  key 
prerequisite  to  support  people  monitoring  at  a  distance  via  sensor  networks.  This  theory  defines  properties  of 
perspective  in  sensor  mediated  experience,  necessary  relationships  between  a  human  and  a  virtual  point-of-view, 
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and  requirements  for  movement  of  the  virtual  point-of-view.  In  the  subsequent  section,  these  actionable  constraints 
and  requirements  are  then  used  to  construct  multiple  instantiations  of  a  perspective  controller. 


Morison,  A.  M.,  Woods,  D.  D.,  &  Davis,  J.  W.  (2009).  How  Panoramic  Visualization  Can  Support  Human 
Supervision  of  Intelligent  Surveillance.  In  In  Proceedings  of  53rd  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society,  San  Antonio,  TX. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  video-based  surveillance  people  monitor  a  wide  spatial  area  through  video  sensors  for  anomalous  events  related 
to  safety  and  security.  The  size  of  the  area,  the  number  of  video  sensors,  and  the  camera's  narrow  field-of-view  make 
this  a  challenging  cognitive  task.  Computer  vision  researchers  have  developed  a  wide  range  of  algorithms  to 
recognize  patterns  in  the  video  stream  (intelligent  cameras).  These  advances  create  a  challenge  for  human 
supervision  of  these  intelligent  surveillance  camera  networks.  This  paper  presents  a  new  visualization  that  has  been 
developed  and  implemented  to  integrate  video-based  computer  vision  algorithms  with  control  of  pan-tilt-zoom 
cameras  in  a  manner  that  supports  the  human  supervisory  role. 


Murphy,  R.  R.,  &  Woods,  D.  D.  (2009).  Beyond  Asimov:  The  Alternative  Laws  of  Robotics.  IEEE  Intelligent 
Systems. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Since  their  codifi  cation  in  1947  in  the  collection  of  short  stories  I,  Robot,  Isaac  Asimov’s  three  laws  of  robotics 
have  been  a  staple  of  science  fi  ction.  Most  of  the  stories  assumed  that  the  robot  had  complex  perception  and 
reasoning  skills  equivalent  to  a  child  and  that  robots  were  subservient  to  humans.  Although  the  laws  were  simple 
and  few,  the  stories  attempted  to  demonstrate  just  how  diffi  cult  they  were  to  apply  in  various  real-world  situations. 
In  most  situations,  although  the  robots  usually  behaved  “logically,”  they  often  failed  to  do  the  “right”  thing, 
typically  because  the  particular  context  of  application  required  subtle  adjustments  of  judgment  on  the  part  of  the 
robot  (for  example,  determining  which  law  took  priority  in  a  given  situation,  or  what  constituted  helpful  or  harmful 
behavior). 


Myers,  B.,  Malkin,  R.,  Bett,  M.,  Waibel,  A.,  Bostwick,  B.,  Miller,  R.  C.,  Yang,  J.,  Denecke,  M.,  Seemann,  E.,  Zhu, 
J.,  Peck,  C.  H.,  Kong,  D.,  Nichols,  J.,  &  Scherlis,  B.  (2003).  Flexi-Modal  and  Multi-Machine  User  Interfaces.  In  In 
Proceedings  of  the  Fourth  IEEE  International  Conference  on  Multimodal  Interfaces  (ICMI’02).  Pittsburgh,  PA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Multi-modal  interfaces,  speech  recognition,  gesture  recognition,  handwriting  recognition,  gaze  tracking,  handhelds, 
personal  digital  assistants  (PDAs),  laser  pointers,  computer  supported  collaborative  work  (CSCW) 

Summary  abstract: 

We  describe  our  system  which  facilitates  collaboration  using  multiple  modalities,  including  speech,  handwriting, 
gestures,  gaze  tracking,  direct  manipulation,  large  projected  touch-sensitive  displays,  laser  pointer  tracking,  regular 
monitors  with  a  mouse  and  keyboard,  and  wirelessly-  networked  handhelds.  Our  system  allows  multiple, 
geographically  dispersed  participants  to  simultaneously  and  flexibly  mix  different  modalities  using  the  right 
interface  at  the  right  time  on  one  or  more  machines.  This  paper  discusses  each  of  the  modalities  provided,  how  they 
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were  integrated  in  the  system  architecture,  and  how  the  user  interface  enabled  one  or  more  people  to  flexibly  use  one 
or  more  devices. 


Nakamura,  M.,  &  Jones,  L.  (2003).  An  Actuator  for  the  Tactile  Vest  -  a  Torso-Based  Haptic  Device.  In  In 
Proceedings  of  1 1th  International  Symposium  on  Haptic  Interfaces  for  Virtual  Environment  &  Teleoperator 
Systems,  pp. 333-339. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Our  goal  is  to  design  a  robust,  versatile  actuator  for  a  Tactile  Vest,  a  wearable  device  that  provides  haptic  feedback 
to  the  torso.  The  technology  we  propose  to  use  as  the  actuator  for  this  device  is  Nitinol,  a  shape  memory  alloy 
(SMA).  Nitinol  is  capable  of  generating  large  stresses  and  strains,  but  is  small  and  flexible  and  able  to  fit  in  the  thin, 
dynamic  workspace  of  a  vest.  In  experimental  tests  conducted  on  the  SMA,  a  pulse  1  A  in  amplitude  and  1  s  in 
duration  produced  an  average  displacement  of  3.7  mm  and  peak  pressures  about  20  times  the  touch  threshold  for  the 
torso.  These  results  indicate  promise  in  using  Nitinol  to  create  a  torso-based  haptic  device  that  can  generate  a  wide 
range  of  stimuli,  yet  is  comfortable  to  the  point  of  being  invisible  to  the  user  until  a  stimulus  occurs. 


Neville,  K.,  Hoffman,  R.  R.,  Linde,  C.,  Elm,  W.  C.,  &  Fowlkes,  J.  (2008).  The  Procurement  Woes  Revisited.  IEEE 
Intelligent  Systems,  72-75. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
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Journal  Article 

Summary  abstract: 

The  set  of  people  who  are  frustrated  every  day  by  badly  designed  information  technology  is  very  large.  So  is  the  set 
of  people  whose  dollars  pay  for  the  badly  designed  technology.  A  conservative  estimate  ranges  in  the  billions  for  the 
cost  of  large-scale  information  systems  that  end  up  collecting  dust  because  they’re  not  properly  humancentered.  Yes, 
billions  and  still  counting — that’s  the  scary  part. 

Within  this  large  set  of  frustrated  customers  (see  the  sidebar  “When  Systems  Development  Neglects  Human 
Considerations”)  is  a  subset  whose  job  it  is  to  do  something  about  this  situation.  That  subset  includes  policymakers, 
program  managers,  and  systems  engineers.  It  also  includes  a  sub-subset  comprising  cognitive  systems  engineers, 
ethnographers,  and  many  others  who,  in  one  vernacular  or  another,  advocate  human-centered  computing.  We  must 
show  that  intelligent  technologies — those  designed  to  interact  with  humans  or  play  a  role  in  the  cognitive  work 
conducted  in  sociotechnical  work  systems — are  usable,  useful,  and  understandable. 


Obradovich,  J.  H.,  Graham,  J.,  Schneider,  M.,  Gonzalez,  C.,  &  Harder,  R.  (2004).  Experimentation  for  Envisioned 
Worlds:  Understanding  the  Limitations  of  the  Military  after  Action  Review  as  Experimental  Data.  In  In  Proceedings 
of  48th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  research  reported  here  was  conducted  to  explore  the  efficacy  of  using  the  military  After  Action  Review  (AAR) 
process  as  a  data  collection  methodology.  During  a  weeklong  U.S.  Army  experiment,  we  investigated,  by  means  of  a 
periodically  administered  questionnaire  and  an  end-of-day  log,  the  interactions  participants  reported  having  with 
other  team  members.  When  comparing  the  results  of  these  tools,  we  found  strong  tendencies  of  the  primacy  and 
recency  effects.  These  findings  are  important  to  researchers  working  in  military  command  and  control 
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experimentation  settings  where  the  AAR  is  a  commonly  used  data  collection  mechanism.  This  paper  identifies  data 
collection  practices  that  can  help  avoid  potential  difficulties  when  using  end-of-day  retrospective  reports  as  an 
experimental  methodology. 


Obradovich,  J.  H.,  &  Smith,  P.  J.  (2003).  Continuous  Distributed  Planning  in  the  Army:  Arriving  at  Design 
Concepts  through  Observational  Studies.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference 
2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of 
Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
observation 

Summary  abstract: 

Coordination  and  collaboration  across  distributed  teams  during  planning,  preparation,  and  execution  activities 
present  many  challenges  to  military  command  and  control.  One  of  these  challenges  is  supporting  distributed  work 
during  continuous  planning  events  that  occur  prior  to  and  during  the  execution  of  an  operation.  This  paper  discusses 
observations  made  during  field  studies  of  exercises  and  experiments  conducted  over  the  past  year,  as  well  as  design 
concepts  arising  out  of  these  observations. 


Obradovich,  J.  H.,  &  Smith,  P.  J.  (2003).  Design  Concepts  for  Distributed  Work  Systems:  An  Empirical 
Investigation  into  Distributed  Teams  in  Complex  Domains.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society  Conference.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Coordination  and  collaboration  across  distributed  teams  during  planning  and  execution  activities  present  many 
challenges  to  organizations  and  operations,  including  military  command  and  control.  With  the  increase  in  spatially 
and  temporally  distributed  work  teams,  it  is  essential  to  achieve  an  understanding  of  how  distributed  work  can  be 
effectively  supported.  In  this  paper,  we  discuss  issues  surrounding  distributed  work  systems  that  were  gained 
through  a  series  of  investigations,  including 

(a)  field  studies, 

(b)  structured  interviews, 

(c)  critical  incident  reports, 

(d)  participation  in  concept  experiments,  and 

(e)  literature  reviews. 

Characteristics  held  in  common  by  the  domains  explored  include  the  following:  high  cognitive  complexity, 
distribution  of  the  work  (and  associated  responsibilities)  among  many  people,  distribution  of  data  and  knowledge 
among  many  people,  uncertainty  about  how  scenarios  will  actually  play  out,  and  geographic  and  temporal 
distribution  of  the  participants.  The  results  of  these  investigations  presented  in  the  form  of  case  studies,  and  design 
concepts  to  support  distributed  work  systems  are  suggested. 


Ogden,  W.  (2009).  Chapter  19.  A  Task-Based  Evaluation  Method  for  Embedded  Machine  Translation  in  Instant 
Messaging  Systems.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter: 
Foundations  and  Technology.  Section  IV.  Communicating  Information  across  the  Team:  48hrBooks  by  Alion 
Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving 
Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision 
Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board 
Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue 
SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 
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Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

Cross-language  instant  messaging  (CL1M)  systems  are  intended  to  be  used  by  groups  of  people  who  do  not  speak 
the  same  language  but  need  to  work  together  to  accomplish  common  tasks.  Such  systems  could  help  coordinate 
teams  and  facilitate  communications  by  automatically  translating  text  messages  sent  to  team  members.  Members 
ofthe  team  would  be  able  to  read  and  enter  messages  in  their  own  languages  and  still  communicate  with  all  members 
of  the  group.  This  could  be  particularly  useful  for  many  multinational  civilian  and  military  missions,  where 
participants  may  not  speak  English  fluently  or  at  all. 

In  this  chapter,  we  describe  a  series  of  user  studies  designed  to  investigate  the  usefulness  ofthese  systems  for  these 
purposes.  These  studies  were  focused  on  three  distinct  goals.  First,  we  wanted  to  develop  a  new  task-based  method 
for  evaluating  the  embedded  machine  translation  (MT)  technology  in  these  systems.  Previous  MT  evaluations  have 
not  been  focused  on  this  type  of  interactive  application.  Second,  we  wanted  to  use  this  method  to  evaluate  the 
usefulness  of  MT  technology  for  instant  messaging  and  suggest  specific  improvements  for  this  application.  Finally, 
we  wanted  to  conduct  formative  evaluations  of  the  user  interface  for  cross-language  instant  messaging  to  improve 
the  user  experience  ofthese  systems.  Consequently,  we  examined  the  habitability  of  the  system  for  accomplishing 
realistic  collaborative  tasks  and  to  communicate  relevant  information  when  interacting  with  collaborators  who  speak 
different  languages.  In  sum,  we  were  interested  in  technical  design  issues  of  both  the  user  interface  and  the  MT 
software  at  the  user  interaction  level. 

In  this  effort  to  date,  we  have  developed  new  paradigms  for  evaluating  machine  translation  technology  for  purposes 
of  collaborative  communication.  We  have  been  conducting  formative  user  testing  of  a  translingual  instant  messaging 
system  interface  (TriM),  developed  by  The  MITRE  Corporation  to  help  military  coalition  partners  to  communicate 
using  their  own  languages  (Miller  et.  al.,  2001).  In  this  paper,  we  will  review  issues  of  conducting  task-based 
evaluations  of  embedded  MT  and  discuss  some  of  the  results  and  implications  for  MT  technology  embedded  in 
CUM.  Next  we  use  these  implications  to  design  and  test  an  improvement  to  the  TriM  user  interface.  Then  we 
describe  a  study  to  test  the  sensitivity  of  the  method  we  have  developed  by  comparing  two  Korean-English 
translation  systems. 


Olson,  L.,  Winslett,  M.,  Tonti,  G.,  Seeley,  N.,  Uszok,  A.,  &  Bradshaw,  J.  (2006).  Trust  Negotiation  as  an 
Authorization  Service  for  Web  Services.  In  In  Proceedings  of  International  Workshop  on  Security  and  Trust  in 
Decentralized/Distributed  Data  Structures  (STD3S)  in  conjunction  with  The  22nd  International  Conference  on  Data 
Engineering  (ICDE'06).  Atlanta,  GA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Like  other  open  computing  environments,  web  services  need  a  scalable  method  of  determining  authorized  users.  We 
present  desiderata  for  authorization  facilities  for  web  services,  and  analyze  potential  ways  of  satisfying  them.  We 
propose  a  third-party  authorization  system  for  web  services  based  on  trust  negotiation,  discuss  its  implementation 
using  the  TrustBuilder  runtime  system  for  trust  negotiation,  and  present  performance  results  from  a  stock  trading 
application. 


Oron-Gilad,  T.  (2005).  Interface  Design:  A  Cognitive  Systems  Engineering  Perspective.  Interfaces  for  Ground  and 
Air  Military  Robots:  Workshop  Summary.  In  Washington,  DC:  The  National  Academies  Press.  ISBN:  0-309- 
54919-1.  http://www.nap.edu/catalog/11251.html  (pp.  13-15). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  the  early  years  of  robotics  and  automated  vehicles,  the  fight  was  against  nature  and  not  against  a  manifestly 
intelligent  opponent.  In  this  context,  researchers  and  engineers  in  artificial  intelligence  aspired  to  design  completely 
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autonomous  systems.  In  military  environments,  however,  where  prediction  and  anticipation  are  complicated  by  the 
existence  of  an  intelli-  gent  adversary,  it  is  essential  to  retain  human  operators  in  the  control  loop. 

Future  military  and  civilian  interface  technologies  will  be  influenced  greatly  by  currently  evolving  autonomous  and 
semiautonomous  systems.  Operators  will  act  at  times  as  monitors,  as  controllers,  and  as  supervisors,  each  role 
putting  different  and  new  demands  on  their  perceptual,  motor,  and  cognitive  capacities.  In  particular,  future  combat 
systems  will  require  operators  to  control  and  monitor  aerial  and  ground  robotic  systems  and  to  act  as  part  of  larger 
teams  coordinating  diverse  robotic  systems  over  mul-  tiple  echelons.  The  goals  for  future  operator  control  units  are 
that  they  be  (1)  integrated  into  the  soldier’s  total  task  environment,  (2)  capable  of  being  used  to  monitor  and  control 
multiple  systems,  and  (3)  interchangeable,  with  a  minimum  of  practice,  among  nonexpert  soldiers.  Display  designers 
should  give  consideration  to  the  trade-offs  between  meeting  diverse  opera-  tional  requirements  and  minimizing 
display  and  control  requirements. 

In  this  context,  in  November  2004  the  National  Research  Council’s  Committee  on  Human  Factors  hosted  a 
workshop  with  support  from  the  Army  Research  Laboratory.  The  workshop  addressed  the  challenges  of  op-  erator 
control  unit  design  from  the  perspectives  of  engineers  who  design  these  interfaces  and  human  factors  specialists  who 
provide  design  guidance  based  on  an  understanding  of  human  cognitive  and  physical  capabilities  and  limitations. 

The  workshop  was  focused  on  the  operational  context  at  the  brigade  level  and  below.  Physical  design  issues  ranged 
from  such  prob-  lems  as  the  size,  resolution,  and  control  constraints  for  small  infantry-  carried  operator  control  units 
to  the  impact  of  motion  for  an  operator  con-  trol  unit  mounted  on  large  armored  vehicles.  Cognitive  requirements  re¬ 
lated  to  design  issues  include  the  need  to  support  common  situation  awareness,  multitasking,  and  teaming  across 
multiple  echelons.  The  major  goal  of  the  workshop  was  to  identify  the  most  important  human-related  research  and 
design  issues  from  both  the  engineering  and  human  factors  perspectives  and  to  develop  a  list  of  lessons  learned  and 
fruitful  research  directions. 

The  workshop  planning  committee  was  composed  of  three  members  of  the  National  Research  Council’s  Committee 
on  Human  Factors — Peter  Hancock  (workshop  chair),  University  of  Central  Florida,  John  Lee,  Uni-  versity  of  Iowa, 
and  Joel  Warm,  University  of  Cincinnati.  This  committee  was  responsible  for  organizing  the  workshop  and 
suggesting  topics  and  pre-  senters.  Leading  engineers  and  human  factors  researchers  were  chosen  based  on  their 
work  on  human-robotic  interface  design. 

The  specific  objectives,  introduced  at  the  workshop  by  Michael  Barnes  of  the  Army  Research  Laboratory,  included: 

•  Examining  scalable  interfaces  in  terms  of  specialized  requirements,  3-D  displays,  multimodal  displays,  handheld 
versus  mounted  control  units,  operational  constraints,  and  common  look  and  feel. 

•  Summarizing  “what  we  know”  and  “what  we  need  to  know”  as  a  precursor  for  design  guidelines. 

•  Identifying  research  areas  ripe  for  exploitation. 

It  is  evident  from  the  presentations  and  discussions  that  took  place  throughout  the  workshop  that  work  on  scalability 
is  in  its  infancy.  Rather  than  presenting  definitive  design  solutions,  presentations  focused  on  such  research  topics  as 
span  of  control,  communication  paradigms,  multitasking,  levels  of  detail,  and  size  of  display  or  control  units.  These 
topics  represent  different  elements  that  contribute  to  an  overall  understanding  of  interface  scalability  issues. 


Pantaleev,  A.,  &  Josephson,  J.  (2006).  Higher-Level  Fusion  for  Military  Operations  Based  on  Abductive  Inference: 
Proof  of  Principle.  In  The  International  Society  for  Optical  Engineering  (SPIE)  Defense  and  Security  Symposium, 
Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
abduction,  abductive  inference,  best-explanation  reasoning,  multisource  information  fusion,  multisensor  fusion, 
higher-level  fusion,  priority  intelligence  requirements,  PIR,  JDL  level  2,  JDL  level  3 

Summary  abstract: 

The  ability  of  contemporary  military  commanders  to  estimate  and  understand  complicated  situations  already  suffers 
from  information  overload,  and  the  situation  can  only  grow  worse.  We  describe  a  prototype  application  that  uses 
abductive  inferencing  to  fuse  information  from  multiple  sensors  to  evaluate  the  evidence  for  higher-level  hypotheses 
that  are  close  to  the  levels  of  abstraction  needed  for  decision  making  (approximately  JDL  levels  2  and  3).  Abductive 
inference  (abduction,  inference  to  the  best  explanation)  is  a  pattern  of  reasoning  that  occurs  naturally  in  diverse 
settings  such  as  medical  diagnosis,  criminal  investigations,  scientific  theory  formation,  and  military  intelligence 
analysis.  Because  abduction  is  part  of  common-sense  reasoning,  implementations  of  it  can  produce  reasoning  traces 
that  are  very  human  understandable.  Automated  abductive  inferencing  can  be  deployed  to  augment  human 
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reasoning,  taking  advantage  of  computation  to  process  large  amounts  of  information,  and  to  bypass  limits  to  human 
attention  and  short-term  memory. 

We  illustrate  the  workings  of  the  prototype  system  by  describing  an  example  of  its  use  for  small-unit  military 
operations  in  an  urban  setting.  Knowledge  was  encoded  as  it  might  be  captured  prior  to  engagement  from  a  standard 
military  decision  making  process  (MDMP)  and  analysis  of  commander's  priority  intelligence  requirements  (PIR). 
The  system  is  able  to  reasonably  estimate  the  evidence  for  higher-level  hypotheses  based  on  information  from 
multiple  sensors.  Its  inference  processes  can  be  examined  closely  to  verifY  correctness.  Decision  makers  can 
override  conclusions  at  any  level  and  changes  will  propagate  appropriately. 


Pantaleev,  A.,  &  Josephson,  J.  (2007).  Prospects  for  Dynamic  ISR  Tasking  and  Interpretation  Based  on  Standing 
Orders  to  Sensor  Networks.  In  In  Proceedings  of  Conference  on  Multisensor,  Multisource  Information  Fusion: 
Architectures,  Algorithms,  and  Applications,  part  of  The  International  Society  for  Optical  Engineering  (SPIE) 
Defense  and  Security  Symposium.  25th  Army  Science  Conference,  Orlando,  FL.,  and  Conference  on  Multisensor, 
Multisource  Information  Fusion:  Architectures,  Algorithms,  and  Applications,  part  of  The  International  Society  for 
Optical  Engineering  (SPIE)  Defense  and  Security  Symposium. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
standing  orders,  PIRs,  automated  fusion,  sensor  networks,  multisensor  fusion 

Summary  abstract: 

This  research  is  intended  to  contribute  to  the  development  of  automated  and  human-in-the-loop  systems  for  higher 
level  fusion  to  respond  to  the  information  requirements  of  command  decision  making.  In  tactical  situations  with 
short  time  constraints,  the  analysis  of  information  requirements  may  take  place  in  advance  for  certain  classes  of 
problems,  and  provided  to  commanders  and  their  staff  as  part  of  the  control  and  communications  systems  that  come 
with  sensor  networks.  In  particular,  it  may  be  possible  that  certain  standing  orders  can  assume  the  role  of  Priority 
Intelligence  Requirements.  Standing  orders  to  a  sensor  network  are  analogous  to  standing  orders  to  Soldiers.  Trained 
Soldiers  presumably  don't  need  to  be  told  to  report  contact  with  hostiles,  for  example,  or  to  report  any  sighting  of 
civilians  with  weapons.  Such  standing  orders  define  design  goals  and  engineering  requirements  for  sensor  networks 
and  their  control  and  inference  systems.  Since  such  standing  orders  can  be  defined  in  advance  for  a  class  of 
situations,  they  minimize  the  need  for  situation-specific  human  analysis.  Thus,  standing  orders  should  be  able  to 
drive  automatic  control  of  some  network  functions,  automated  fusion  of  sensor  reports,  and  automated  dissemination 
of  fused  information.  We  define  example  standing  orders,  and  outline  an  algorithm  for  responding  to  one  of  them 
based  on  our  experience  in  the  field  of  multisensor  fusion. 


Park,  S.,  &  Rothrock,  L.  (2008).  Systematic  Analysis  of  Framing  Bias  in  Missile  Defense:  Implications  toward 
Visualization  Design.  European  Journal  of  Operational  Research  Volume  182,  Issue  3,  Pages  1383-1398. 
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Decision  analysis;  OR  in  military;  Utility  theory 
Summary  abstract: 

This  paper  examines  the  effects  of  framing  on  decision  making  in  a  homeland  missile  defense  context  across  three 
tasks  of  varying  complexity.  Mathematically,  each  task  was  modeled  as  abstractions  from  a  common  resource 
allocation  task.  Logically,  therefore,  the  effects  of  framing  on  human  subjects  should  have  been  consistent  across  all 
the  tasks.  In  the  first  experiment,  a  simple  lottery  was  used  to  determine  risk  postures  in  a  single-attribute  case  of 
missile  defense.  Results  showed  that,  consistent  with  Prospect  Theory,  positive  framing  promotes  risk-averse 
behavior  whereas  negative  framing  promotes  risk-seeking  behavior.  In  the  second  experiment,  we  used  the  Analytic 
Hierarchy  Process  to  determine  subject  rankings  in  a  multi-attribute  case  of  missile  defense.  Results  suggest  that 
subjects’  performances  under  positive  framing  were  significantly  better  than  performances  under  negative  framing. 
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In  the  third  experiment,  we  used  a  human-in-the-loop  simulation  to  elicit  human  decisions  in  a  missile  defense 
resource  allocation  task.  In  comparison  to  the  other  experiments,  the  framing  effect  in  the  third  experiment  was 
diminished.  We  submit  that  decision  biases  detected  in  a  simple  choice  task  cannot  be  assumed  to  carry  over  to  tasks 
of  greater  complexity  even  if  the  underlying  mathematical  formulation  for  all  the  tasks  is  the  same.  Moreover,  we 
submit  that  the  design  of  the  graphical  interface  has  a  greater  influence  on  human  judgment  bias  than  framing  in 
tasks  of  higher  complexity. 


Patterson,  E.  S.,  Schweikhart,  S.,  Anders,  S.,  &  Render,  M.  L.  (2007).  Towards  a  Functional  Model  of  Quality 
Improvement  Collaboratives.  In  In  Proceedings  of  51st  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Baltimore,  MD. 

Key  Words: 
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Summary  abstract: 

Quality  improvement  collaboratives  (QIC)  are  widely  used  for  seeking  improvements  in  healthcare  quality  and 
safety.  Nevertheless,  the  effectiveness  of  QICs  is  variable.  In  order  to  support  research  that  identifies  critical 
elements  in  running  a  successful  collaborative,  we  fill  a  conceptual  gap  by  moving  towards  a  functional  model  of 
QICs.  Specifically,  we  define  how  QICs  are  distinct  from  traditional  quality  improvement  teams,  conceptualize  how 
primary  and  secondary  functions  are  accomplished  in  a  means-ends  framework,  and  illustrate  how  the  functions  are 
dynamically  accomplished  in  a  series  of  meetings  by  nested  teams  within  a  collaborative.  Finally,  we  discuss 
distinctions  among  QICs. 


Patterson,  E.  S.,  Woods,  D.  D.,  Cook,  R.  I.,  &  Render,  M.  L.  (2005).  Collaborative  Cross-Checking  to  Enhance 
Resilience.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 
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Summary  abstract: 

There  has  been  a  longstanding  consensus  that  supporting  error  detection  and  recovery  processes  is  critical  for  very 
high  safety  levels  because  it  increases  system  resilience.  System  resilience  is  defined  in  Resilience  Engineering  as 
successful  adaptation  to  variations,  changes,  and  surprises  by  organizations,  groups,  or  individuals.  Cross-checking 
is  a  critical  component  of  resilience  because  it  can  enable  detection  of  erroneous  assessments  or  actions  while 
negative  consequences  can  be  mitigated  or  eliminated.  Prior  studies  suggest  that  cross-checking  where  an  additional 
human  with  a  fresh  perspective  breaks  fixations  may  be  an  effective  strategy.  Nevertheless,  collaborative  cross¬ 
checking  remains  a  somewhat  murky  concept.  In  this  paper,  we  describe  in  detail  three  healthcare  incidents  where 
collaborative  cross-checking  played  a  key  role.  Emerging  patterns  that  provide  opportunities  for  follow-on  research 
are  discussed. 


Pazuchanics,  S.  L.  (2004).  The  Effect  of  Combining  Monaural  Cues  to  Distance.  In  In  Proceedings  of  48th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA.  (pp.  1832-1879). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  degree  to  which  distance  can  be  perceived  depends  on  combinations  of  distance  cues  that  are  inherently 
ambiguous  in  isolation.  Previous  research  on  combining  distance  cues  has  focused  on  static  monocular  cues  and  has 
shown  an  additive  relation.  The  present  experiment  was  designed  to  examine  how  people  integrate  monaural  cues 
(loudness,  delay  of  reception  from  initiation,  and  frequency  attenuation)  to  perceive  distance.  Monaural  distance 
information  was  presented  to  participants  using  desktop  computers  and  headphones.  A  general  linear  model  (GLM) 
analysis  revealed  that  monaural  cues,  much  like  visual  cues,  are  combined  linearly  to  form  a  percept  of  distance,  but 
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with  greater  weighting  given  to  loudness.  Thus,  designers  of  spatial  displays  that  might  be  used  in  conditions  of  poor 
visibility  could  supply  auditory  cues  for  distance  information.  Additionally,  this  paper  could  be  seen  as  a  step  along 
the  way  in  understanding  how  people  integrate  multiple  sources  of  information  to  perceive  distance. 


Pazuchanics,  S.  L.  (2006).  The  Effects  of  Camera  Perspective  and  Field  of  View  on  Performance  in  Teleoperated 
Navigation.  In  In  Proceedings  of  50th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  San 
Francisco  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Directly  teleoperated  uninhabited  ground  vehicles  (UGVs)  can  grant  field  teams  access  to  places  and  information 
previously  unavailable.  Despite  their  capability,  the  video  from  these  vehicles  can  make  teleoperated  navigation 
difficult  by  cutting  out  valuable  information  about  the  spatial  context  around  the  UGV.  Typically,  UGV  cameras 
provide  their  operators  with  a  very  narrow,  “soda  straw”  field  of  view  (FOV)  and  a  first-person  camera  perspective. 
The  present  research  investigated  two  methods  for  providing  an  operator  with  additional  contextual  information: 
widening  the  FOV  and  capturing  a  third-person  perspective  of  the  vehicle  in  its  environment.  The  results  indicate 
that  the  additional  information  provided  by  either  method  can  facilitate  aspects  of  navigation  performance.  Of  the 
two  methods,  widening  the  FOV  produced  the  greatest  performance  benefit  in  our  task.  However,  if  the  FOV  cannot 
be  widened,  the  results  suggest  that  capturing  a  third-person  perspective  may  facilitate  certain  aspects  of  navigation. 
Finally,  the  benefits  associated  with  each  method  were  found  to  be  additive.  This  result  suggests  that  ideal  video 
displays  may  incorporate  both  methods,  but  further  research  is  necessary  to  understand  how  the  specific  information 
provided  by  each  method  impacts  the  many  types  of  tasks  involved  in  teleoperated  navigation. 


Pazuchanics,  S.  L.,  Chadwick,  R.  A.,  Sapp,  M.  V.,  &  Gillan,  D.  J.  (2010).  Chapter  6.  Robots  in  Space  and  Time: 

The  Role  of  Object,  Motion  and  Spatial  Perception  in  the  Control  and  Monitoring  of  UGVs.  In  M.  Barnes  and  F. 
Jentsch  (Eds.),  Human-Robot  Interactions  in  Future  Military  Operations:  New  Mexico  State  University. 
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Summary  abstract: 

The  role  of  object,  motion  and  spatial  perception  in  the  control  and  monitoring  of  UGVs 

Uninhabited  ground  vehicles  (UGVs)  have  the  potential  to  expand  human  presence  and  intent  across  space  and  time 
(Riley,  Murphy,  and  Endsley,  2006).  Through  the  use  of  cameras,  microphones,  grapplers,  and  basic  vehicle 
platforms,  UGVs  can  grant  human  operators  the  basic  abilities  to  “see,”  “manipulate,”  and  “move  through”  off-site 
environments.  Building  on  these  basic  abilities,  UGVs  can  be  applied  to  perform  any  number  of  complicated  tasks 
remotely  in  environments  ill-suited  for  humans.  Because  of  this,  UGVs  have  been  increasingly  incorporated  into  a 
variety  of  dangerous  task  domains  including:  urban  search  and  rescue  (USAR),  explosive  ordinance  disposal  (EOD), 
military  operations  in  urban  terrain  (MOUT)  and  space  exploration. 

Although  the  potential  to  perform  a  number  of  complicated  tasks  remotely  is  certainly  appealing,  success  in  such 
operations  is  largely  reliant  on  a  human’s  ability  to  control  a  UGV’s  actions  and  to  monitor  the  stream  of  data  the 
vehicle  provides  about  its  environment  throughout  the  course  of  its  operation.  Controlling  a  UGV  often  involves 
directing  the  vehicle’s  course  across  a  remotely-located  space  (i.e.  course  control  activities),  and/or  directing  how 
the  vehicle  manipulates  objects  using  devices  such  as  a  grappler  (i.e.  manipulation  control  activities).  Monitoring  a 
UGV  involves  actively  surveying  the  UGV’s  environment  over  the  span  of  its  operation,  often  via  a  video  camera, 
to  determine  the  presence,  motion  and  location  of  objects  of  interest  as  well  as  the  UGV  itself  (i.e.  monitoring 
activities).  While  some  systems  automate  control  and  monitoring  activities,  most  systems  in  current  use  require  a 
“human  in  the  loop”  for  supervision  and  guidance  -  a  situation  unlikely  to  change  (Fong  and  Thorpe,  2001). 
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Despite  the  importance  of  human  control  and  monitoring  for  successful  remote  operation,  these  activities  remain 
notoriously  difficult  across  different  task  domains.  Course  control  tends  to  be  problematic.  For  example,  avoiding 
obstacles  in  a  simulated  USAR  scenario  was  observed  as  a  common  difficulty  across  different  teams  and  systems 
during  the  2002  American  Association  for  Artificial  Intelligence  Robot  Rescue  Competition  (Yanco,  Drury  and 
Scholtz,  2004).  Manipulation  control  is  also  difficult.  Extensive  training  and  workload  are  necessary  to  use  grappler 
arms  to  move  dangerous  objects  in  current  EOD  and  firefighting  systems  (Lundburg  2007).  Monitoring,  too,  can  be 
met  with  considerable  difficulty.  In  USAR  scenarios,  multiple  human  operators  are  often  needed  to  monitor  video 
for  objects  of  interest  within  the  environment  and  to  determine  their  location  (Riley  and  Endsley,  2004). 

To  remedy  control  and  monitoring  problems  such  as  these,  the  role  that  critical  perceptual  processes  play  in  guiding 
everyday  control  and  monitoring  activities  needs  to  be  considered  (Tittle,  Roesler  and  Woods,  2002;  Chen,  Haas, 
Pillalamarri,  and  Jacobson,  2006;  Chen,  Haas  and  Barnes,  2007).  When  humans  control  and  monitor  their  own 
actions  across  space  and  time,  fundamental  perceptual  processes  including,  but  certainly  not  limited  to,  object 
recognition,  motion  awareness  and  the  formation  of  global  spatial  comprehension  allow  these  activities  to  be 
completed  with  some  degree  of  skill.  By  identifying  and  addressing  the  types  of  “remote  perception  problems” 
(Tittle  et  al,  2002)  human  operators  experience  in  UGV  operation  scenarios  and  the  user-interface  and 
environmental  contributors  to  these  problems,  controlling  and  monitoring  these  vehicles  can  be  facilitated.  To  assist 
in  this  endeavor,  this  chapter  describes  the  roles  that  object  recognition,  motion  awareness,  and  global  spatial 
comprehension  play  in  both  everyday  and  UGV  control  and  monitoring  activities.  It  describes  the  difficulties  UGV 
operators  encounter  when  remotely  engaged  in  each  of  these  perceptual  processes  and  discusses  the  potential  factors 
that  contribute  to  these  difficulties.  Finally,  the  chapter  provides  guidance  for  supporting  and  enhancing  remote 
perception  and  proposes  suggestions  for  future  research. 


Pazuchanics,  S.  L.,  &  Gillan,  D.  J.  (2003).  The  Effect  of  Combining  Monaural  Cues  to  Depth  within  and  across 
Modalities.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology 
Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi, 
Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

The  degree  to  which  depth  can  be  perceived  depends  on  the  presence  of  depth  cues,  including  monocular  and 
monaural  depth  cues.  These  cues  are  ambiguous  by  nature.  Any  individual  cue  might  be  interpreted  as  providing 
depth  information  or  some  other  sort  of  information  about  the  environment.  Two  experiments  were  designed  to 
examine  how  people  integrate  monocular  and  monaural  cues  to  gain  a  perception  of  distance  by,  first,  exploring  how 
monaural  cues  affect  this  perception  and,  second,  exploring  how  both  monaural  and  monocular  cues  affect  this 
perception.  Monaural  and  monocular  depth  information  was  presented  to  participants  using  desktop  computers  and 
headphones.  A  general  linear  model  (GLM)  analysis  revealed  that  monaural  cues,  on  their  own,  are  combined 
linearly  to  form  a  percept  of  depth.  However,  when  monaural  and  monocular  information  were  presented 
simultaneously,  visual  cue  dominance  occurs  in  which  monocular  cue  information  overrides  that  provided  by 
monaural  cues.  This  information  can  be  used  to  improve  the  interpretability  of  apparent  depth  displays. 


Pazuchanics,  S.  L.,  &  Gillan,  D.  J.  (2005).  Displaying  Distance  in  Computer  Systems:  A  Lesson  from  Two- 
Dimensional  Works  of  Art.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Orlando,  FL. 
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Summary  abstract: 

Virtual  depth  displays  depend  on  static,  monocular  cues.  Models  of  integrating  monocular  cues  may  be 
continuous  (additive)  or  discontinuous.  Previous  research  using  simple  displays  and  a  small  number  of 
cues  supported  continuous  cue  integration.  The  present  research  is  designed  to  expand  the  understanding 
of  how  the  visual  system  integrates  information  from  multiple  pictorial  cues  by  investigating  combinations 
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of  one  to  ten  pictorial  cues  in  visually-rich,  two-dimensional  displays  (paintings  and  photographs). 
Participants  estimated  depth  in  target  paintings  and  photographs  relative  to  a  standard  two  dimensional 
display.  Certain  results  suggest  that  the  visual  system  integrates  cues  in  a  largely  additive  way,  but  after  a 
number  of  cues  are  present  there  may  be  an  additional  boost  in  perceived  depth  resulting  in  a  best-fitting 
discontinuous  model  of  cue  combination.  However,  this  discontinuous  effect  may  be  due  to  design 
decisions  made  by  the  painters  rather  than  exclusively  to  the  perceptual  processes  of  the  viewers.  Analyses 
of  these  design  decisions  provide  lessons  for  the  design  of  two-dimensional  displays. 


Peffer,  J.  E„  Tittle,  J.  S.,  Gualtieri,  J.  W.,  Elm,  W.  C.,  Voshell,  M.,  Pme,  B„  &  Woods,  D.  D.  (2008).  How  Costly  Is 
Your  C2  Coordination?  Assessing  the  Coordination  Requirements  within  Command  and  Control.  In  In  Proceedings 
of  13th  International  Command  and  Control  Research  and  Technology  Symposium:  C2  for  Complex  Endeavors. 
Bellevue,  WA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
C2  Metrics  and  Assessment;  Cognitive  and  Social  Issues;  C2  Concepts,  Theory  and  Policy 

Summary  abstract: 

As  the  size  and  complexity  of  Command  and  Control  (C2)  domains  continues  to  increase,  new  approaches  (e.g., 
Net-Centric  Warfare)  have  been  proposed  to  dynamically  recruit  additional  resources  by  creating  distributed  teams 
to  handle  rapidly  changing  situations.  Although  the  added  flexibility  does  have  the  potential  to  create  C2  structures 
capable  of  responding  to  a  wide  array  of  challenges,  this  addition  of  distributed  team  members  creates  new 
coordination  costs.  The  performance  of  the  C2  team  is  impacted  by  the  additional  cognitive  demands  now  required 
for  team  coordination  and  collaboration,  in  addition  to  the  fundamental  decisions  of  the  domain. 

Recently,  Voshell,  Woods,  Pme,  and  Fern  (2007)  have  proposed  Coordination  Loops  as  a  means  of  specifying  the 
requirements  for  supporting  distributed  cognitive  work.  This  paper  will  detail  an  extension  of  Coordination  Loops 
from  a  functional  perspective,  to  understand  the  relationship  between  coordination  needs  and  the  stress  they  place  on 
specific  decisions  within  a  work  domain  (e.g.  Emergency  Management).  By  mapping  organizationally  defined  roles 
onto  a  functional,  goal-means  decomposition  of  the  domain,  both  the  inherent  information  needs  of  the  domain  and 
the  additional  information  needs  for  coordination  can  be  understood  in  the  same  context.  This  approach  yielded 
unique  insights  were  gathered  about  coordination  assessment  and  design  to  reduce  cognitive  and  coordination 
workload. 


Pezdek,  K.,  Deffenbacher,  K.  A.,  Lam,  S.,  &  Hoffman,  R.  R.  (2006).  Chapter  11.  Cognitive  Psychology: 
Applications  and  Careers.  In  S.  Donaldson,  D.  Berger,  &  K.  Pezdek  (Eds.),  Applied  Psychology:  New  Frontiers  and 
Rewarding  Careers  (pp.  221-241).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates,  Inc. 
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Summary  abstract: 

Psychological  research  is  often  conceptualized  as  being  either  basic  research  or  applied  research.  Basic 
psychological  research,  usually  laboratory  based,  is  conducted  to  gain  a  scientific  understanding  of  human  behavior, 
often  primarily  for  the  purpose  of  advancing  relevant  theories.  Applied  psychological  research  is  conducted  for 
some  additional  reasons:  to  gain  a  scientific  understanding  of  human  behaviors  that  occur  in  work  and  everyday  life, 
or  to  solve  practical  problems  encountered  in  daily  pursuits.  In  this  chapter,  historical  and  current  trends  in  cognitive 
psychology  research  are  presented  to  demonstrate  the  rich  tradition  of  applied  research  in  this  specific  field  of 
psychological  research. 

Cognitive  psychology  is  the  study  of  human  intelligence  and  how  it  works.  This  includes  the  study  of  thinking, 
reasoning,  problem  solving,  decision  making,  memory,  language,  perception,  and  attention.  Cognitive  psychology 
research  is  well  known  for  its  strong  theoretical  base.  Whether  inductively  (theory  is  derived  from  data)  or 
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deductively  (the  theory  comes  first  and  the  empirical  test  follows),  cognitive  psychology  research  is  strongly  linked 
to  theory.  However,  it  is  an  oversimplification  to  categorize  cognitive  psychology  as  basic  research,  from  which  few 
applications  have  been  derived.  Furthermore,  it  is  a  fact  that  many  cognitive  psychologists  work  in  industry, 
educational  settings,  and  the  private  sector  (e.g.,  human  factors  consulting  firms).  From  psychology's  early  days  to 
the  present,  there  has  been  an  interest  in  applications.  The  first  President  of  the  American  Psychological  Association 
(APA),  G.  Stanley  Hall,  conducted  applied  research,  especially  on  topics  in  human  development.  Few  realize  that  he 
was  also  the  founder  of  the  APA  journal.  Journal  of  Applied  Psychology  in  1917.  The  rich  history  of  research  in 
applied  cognitive  psychology  is  reviewed  in  more  detail  by  Hoffman  and  Deffenbacher  (1992). 

The  primary  purpose  of  this  chapter  is  to  dispel  the  myth  that  cognitive  psychology  research  is  primarily  basic 
research  that  has  little  applicability  to  real-world  problems.  This  chapter  briefly  focuses  on  five  domains  of  applied 
cognitive  psychology.  Each  of  these  is  a  domain  in  which  there  is  a  rich  history  of  research  and  in  which  a 
significant  number  of  cognitive  psychologists  are  currently  employed  outside  of  academia.  The  domains  to  be 
discussed  are  forensic  psychology,  marketing  and  advertising,  education,  the  military,  and  human  factors.  Within 
each  domain,  a  brief  history  of  cognitive  psychology  research  on  this  topic  is  discussed,  some  of  the  current  work  on 
this  topic  is  presented,  and  examples  of  careers  in  this  field  are  provided. 


Phillips,  F.  (2007).  Supporting  Research  through  the  Art  of  Visual  Storytelling.  Podcast.  Http://Csel. Eng. Ohio- 
State.  Edu/Podcasts/Flip/Story.Html.  In  Podcast,  http://csel.eng.ohio-state.edu/podcasts/flip/story.html. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Good  science  consists  of  more  than  analytic  integration  and  testing  of  concepts.  Compelling  communication  is 
needed  so  that  the  relevant  ideas  are  not  lost  in  the  'noise'  of  data,  technology  and  cha  nge.  Clearly,  this  can  work 
both  ways-  weak  concepts  and  poor  analysis  can  be  ‘over-presented.'  To  deal  with  this  conundrum  all  students  of 
science  would  be  well  served  by  learning  and  using  visual  story  telling  skills. 

As  part  of  the  Institute  for  Collaborative  Innovation  run  by  Ohio  State  University's  Cognitive  Systems  Engineering 
Laboratory  (CIS/E/L),  Professor  Phillips  speaks  on  storyboarding  as  a  technique  to  evolve  narrative  support  and 
structure  for  analysis  as  well  as  for  communication.  This  enhanced  podcast  complements  on  going  C/S/E/L  work 
exploring  new  techniques  In  the  creative  design  process  and  animated  scenario  development. 

Professor  Flip  Phillips  of  Skidmore  has  participated  in  the  annual  Institute  for  Collaborative  Innovation  as  part  of 
collaborations  with  C/S/E/L  team  members.  He  combines  research  human  perception  and  his  experience  as  an 
animation  scientist  and  technical  director  at  Pixar  Animation  Studios  to  innovate  new  forms  of  human-robot 
coordination 


Phillips,  F.,  &  Voshell,  M.  (2007).  A  Novel  Metric  for  Evaluating  Human-Robot  Navigation  Performance.  In  In 
Proceedings  of  RTA/HFM  Symposium:  Human  Factors  of  Uninhabited  Military  Vehicles  as  Force  Multipliers, 
Biarritz,  France. 
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H.4  [Information  Systems  Applications]:  Miscellaneous; 

D.2.8  [Software  Engineering]:  Metrics-complexity  measures,  performance  measures 
General  Terms 

fractal  dimension.  Tortuosity,  Human  Robot  Coordination 
Summary  abstract: 

It  is  frequently  desirable  to  quantitatively  evaluate  performance  in  human-robot  interaction.  In  this  research,  we 
introduce  a  novel  use  of  fractal  based  performance  measures  to  evaluate  robot  navigation.  It  can  be  difficult  to  use 
simple  measurements  such  as  path  length  and  task  timing  due  to  large  amounts  of  individual  variability  along  these 
metrics.  This  shortcoming  makes  these  measures  particularly  unsuitable  for  evaluating  human/robot  navigation 
performance  in  a  complex  environment  with  multiple  tasks.  Herein  we  outline  a  class  of  novel  metrics  for  evaluating 
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navigation  performance  using  fractal  geometry.  This  class  of  space  and  time  invariant  metrics  allows  us  to  better 
analyze  humau-robot  exploration,  interpret  valuable  behavioral  information  for  analysis  of  movement,  measure  the 
handler/robot's  search  efficiency,  path  tortuosity,  and  overall  space  utilization  in  relation  to  handler  goals  and  overall 
characteristics  of  the  environment.  We  present  techniques  for  computation  and  empirical  results  from  our  own 
navigation  interface  studies. 


Pierce,  L.  G.  (2002).  Barriers  to  Adaptability  in  a  Multinational  Team.  In  In  Proceedings  of  46th  Annual  Meeting  of 
the  Human  Factors  and  Ergonomics  Society.  Baltimore,  MD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  majority  of  current  military  missions  require  collaboration  among  multiple  nations.  For  example,  the  military 
presence  established  in  Bosnia-Herzegovina  (B-H)  following  the  end  of  their  civil  war  included  participants  from 
over  38  nations.  U.S.  Army  forces  were  assigned  to  the  headquarters  element  and  to  Multinational  Division  North 
(MND(N)).  We  reviewed  processes  of  the  U.S.  Army  force  commanding  MND(N)  to  assess  the  degree  to  which 
army  forces  were  being  prepared  to  adapt  their  warfighting  skills  to  conduct  multinational  peacekeeping  missions. 
Barriers  to  adaptability  were  identified  in  their  approach  to  deployment  training  and  the  organization  of  the  military 
headquarters.  U.S.  forces  were  not  trained  to  work  with  multinational  partners  and  multinational  staff  members  were 
not  integrated  into  staff  planning  processes.  Methods  to  improve  team  adaptability  and  a  framework  for  considering 
the  relationship  among  cultural,  social  cognitive  processes  and  multinational  teamwork  are  proposed. 


Pisu,  P.,  Cantemir,  C.-G.,  Dembski,  N.,  Rizzoni,  G.,  Serrao,  L.,  Josephson,  J.  R.,  &  Russellc,  J.  (2006).  Evaluation 
of  Powertrain  Solutions  for  Future  Tactical  Truck  Vehicle  Systems.  In  The  International  Society  for  Optical 
Engineering  (SPIE)  Defense  and  Security  Symposium,  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
military  truck,  hybrid  powertrain,  design  space  exploration,  simulation,  optimization. 

Summary  abstract: 

The  article  presents  the  results  of  a  large  scale  design  space  exploration  for  the  hybridization  of  two  off-road 
vehicles,  part  of  the  Future  Tactical  Truck  System  (FTTS)  family:  Maneuver  Sustainment  Vehicle  (MSV)  and 
Utility  Vehicle  (UV).  Series  hybrid  architectures  are  examined.  The  objective  of  the  paper  is  to  illustrate  a  novel 
design  methodology  that  allows  for  the  choice  of  the  optimal  values  of  several  vehicle  parameters.  The  methodology 
consists  in  an  extensive  design  space  exploration,  which  involves  running  a  large  number  of  computer  simulations 
with  systematically  varied  vehicle  design  parameters,  where  each  variant  is  paced  through  several  different  mission 
profiles,  and  multiple  attributes  of  performance  are  measured.  The  resulting  designs  are  filtered  to  choose  the  design 
tradeoffs  that  better  satisfy  the  performance  and  fuel  economy  requirements.  At  the  end,  few  promising  vehicle 
configuration  designs  will  be  selected  that  will  need  additional  detailed  investigation  including  neglected  metrics 
like  ride  and  drivability.  Several  powertrain  architectures  have  been  simulated.  The  design  parameters  include  the 
number  of  axles  in  the  vehicle  (2  or  3),  the  number  of  electric  motors  per  axle  (1  or  2),  the  type  of  internal 
combustion  engine,  the  type  and  quantity  of  energy  storage  system  devices  (batteries,  electrochemical  capacitors  or 
both  together).  An  energy  management  control  strategy  has  also  been  developed  to  provide  efficiency  and 
performance.  The  control  parameters  are  tunable  and  have  been  included  into  the  design  space  exploration.  The 
results  show  that  the  internal  combustion  engine  and  the  energy  storage  system  devices  are  extremely  important  for 
the  vehicle  performance. 


Plott,  C.  C.,  Endsley,  M.  R.,  &  Strater,  L.  D.  (2004).  Integrating  Sagat  into  Human  Performance  Models.  In  In 
Proceedings  of  2004  Conference  on  Human  Performance:  Situation  Awareness  and  Automation.  Daytona  Beach, 
FL. 
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Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Situation  awareness;  SAGAT;  Human  performance  modeling 

Summary  abstract: 

Over  the  last  decade  a  number  of  methods  for  empirically  assessing  individual  and  team  situation  awareness  in 
simulator  and  operational  environments  have  been  developed  and  applied.  The  Situation  Awareness  Global 
Assessment  Technique  (SAGAr)  has  been  used  across  a  number  of  operational  contexts  for  empirically  assessing 
situation  awareness.  The  results  of  these  applications  have  shown  that  SAGAT  is  both  theoretically  sound  and 
provides  a  valid  method  for  assessing  situation  awareness.  During  the  same  period,  simulation  models  of  human 
performance  for  assessing  and  predicting  individual  and  team  performance  times,  accuracies,  and  workload  were 
also  developed  and  applied  across  a  wide  variety  of  contexts.  In  particular,  several  efforts  have  demonstrated  the 
predictive  validity  of  task  network  models  of  human  performance.  Measures  of  situation  awareness  have  only 
recently  begun  to  be  incorporated  into  these  types  of  human  performance  models.  In  this  paper,  we  describe  a 
method  for  incorporating  the  SAGA  T  scoring  system  into  task  network  models  of  human  performance.  In  addition, 
we  provide  a  discussion  of  the  benefits  of  exploring  the  dynamics  of  the  situation  awareness  measures  through  the 
exercising  of  the  human  performance  models. 


Potter,  S.  S.,  Woods,  D.  D.,  Roth,  E.  M.,  Fowlkes,  J.,  &  Hoffman,  R.  R.  (2006).  Evaluating  the  Effectiveness  of  a 
Joint  Cognitive  System:  Metrics,  Techniques,  and  Frameworks.  In  In  Proceedings  of  50th  Annual  Meeting  of  the 
Human  Factors  and  Ergonomics  Society.  San  Francisco  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

An  implication  of  Cognitive  Systems  Engineering  is  that  joint  cognitive  systems  (JCS;  also  known  as  complex 
socio-technical  systems)  need  to  be  evaluated  for  its  effectiveness  in  performing  the  complex  cognitive  work 
requirements.  This  requires  using  measures  that  go  well  beyond  ‘'‘typical”  performance  metrics  such  as  the  number 
of  subtask  goals  achieved  per  person  per  unit  of  time  and  the  corresponding  simple  baseline  comparisons  or 
workload  assessment  metrics.  This  JCS  perspective  implies  that  the  system  must  be  designed  and  evaluated  from  the 
perspective  of  the  shift  in  role  of  the  human  supervisor.  This  imposes  new  types  of  requirements  on  the  human 
operator.  Previous  research  in  CSE  and  our  own  experience  has  lead  us  to  identify  a  set  of  generic  JCS  support 
requirements  that  apply  to  cognitive  work  by  any  cognitive  agent  or  any  set  of  cognitive  agents,  including  teams  of 
people  and  machine  agents.  Metrics  will  have  to  reflect  such  phenomena  as  “teamwork”  or  “resilience”  of  a  JCS. 
This  places  new  burdens  on  evaluation  techniques  and  frameworks,  since  metrics  should  be  generated  from  a 
principled  approach  and  based  on  fundamental  principles  of  interest  to  the  designers  of  the  JCS.  An  implication  of 
the  JCS  perspective  is  that  complex  and  cognitive  systems  need  to  be  evaluated  for  usability,  usefulness,  and 
understandability;  each  of  which  goes  well  beyond  raw  performance.  However,  conceptually-grounded  evaluation 
frameworks,  corresponding  operational  techniques,  and  corresponding  measures  for  these  are  limited.  Therefore,  in 
order  to  advance  the  state  of  the  field,  we  have  gathered  a  set  of  researchers  and  practitioners  to  present  recent 
evaluation  work  to  stimulate  discussion. 


Priest,  H.  A.,  Burke,  C.  S.,  Munim,  D.,  &  Salas,  E.  (2002).  Understanding  Team  Adaptability:  Initial  Theoretical  and 
Practical  Considerations.  In  In  Proceedings  of  46th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society. 
Baltimore,  MD.  (pp.  561-565). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Team  adaptability  is  just  beginning  to  be  understood  by  researchers.  Team  training,  team  effectiveness,  and 
adaptability  have  provided  researchers  with  a  pool  of  evidence  that  can  be  leveraged  into  meaningful  team 
adaptability  research.  However,  team  adaptability  is  different  from  individual  adaptability.  Teams  have  processes 
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that  individuals  do  not.  Furthermore,  team  adaptability  refers  to  more  than  just  effective  performance.  The  following 
paper  identifies  2  team  processes  that  past  literature  indicates  is  important  to  adaptability  (feedback  and  shared 
mental  models).  Theoretical  issues  and  practical  training  issues  are  examined  to  help  determine  their  role  in  adaptive 
teams. 


Rehak,  M.,  Pechoucek,  M.  &  Bradshaw,  J.  M.  (2007).  Representing  Context  for  Multiagent  Trust  Modeling.  In  In 
Proceedings  of  2006  IEEE/WIC/ACM  International  Conference  on  Intelligent  Agent  Technology.  Hong  Kong,  . 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

We  present  a  universal  mechanism  that  can  be  combined  with  existing  trust  models  to  extend  their  capabilities 
towards  efficient  modelling  of  the  situational  (contextdependent)  trust.  The  mechanism  describes  the  similarity 
between  the  situations  using  their  distance  in  a  metric  space  and  defines  a  set  of  reference  contexts  in  this  space  to 
which  it  associates  the  trustfulness  data.  The  data  associated  with  each  reference  context  is  updated  and  queried  with 
the  weight  that  decreases  with  distance  between  the  current  situation  and  the  reference  context.  In  the  presented 
mechanism,  we  use  Leader-Follower  clustering  to  place  the  reference  contexts  to  be  representative  of  the  data.  In  an 
empirical  test,  we  show  that  context-aware  models  easily  outperform  the  general  trust  when  the  situation  has  an 
impact  on  partner  trustfulness  and  that  their  performance  and  efficiency  is  comparable  with  general  trust  models 
when  the  trustfulness  is  independent  of  the  situation.  Multi-context  nature  of  the  model  also  expands  its  use  towards 
more  advanced  uses,  allowing  policy/norm  learning  from  at  the  trust  model  at  runtime,  as  well  as  reasoning  based  on 
uncertain  identities. 


Riley,  J.,  &  Strater,  L.  (2006).  Assessing  Effects  of  Robot  Control  Mode  on  Performance  and  Situation  Awareness 
in  a  Maze  Navigation  Task.  In  In  Proceedings  of  3rd  Annual  Workshop  on  Human  Factors  of  Unmanned  Aerial 
Vehicles,  Mesa,  Arizona. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

Situation  awareness  in  HRI 

•  Ability  to  complete  remote  tasks  has  improved  with  more  sophisticated  unmanned  vehicles  (UVs). 

•  UVs  used  to  facilitate  perception  and  project  operator  intent  on  space. 

•  Effective  use  of  UVs  dependent  upon: 

-  User  skills 

-  System  design  and  capabilities 

-  Knowledge  that  user  is  able  to  acquire  for  task  completion 

•  Situation  awareness  (SA)  is  knowing  what’s  going  on  around  you. 

-  Endsley  (1988)  defines  it  as  perception,  comprehension,  and  projection. 

-  For  robotic  system  it’s  awareness  of: 

•  Local  environment 

•  Remote  environment 

•  System  status 

•  Current  task  state  /  robot  actions 

•  Task  requirements 

•  Other  team  members  (robots  and/or  humans) 

-  SA  critical  because  it  is  the  framework  for  plans  and  action,  the  foundation  for  decision  making. 

Current  Objective 

•  Look  into  specific  aspects  of  the  model 

-  Assess  effects  of  robot  control  requirements  (and  the  addition  of  automation)  on  operator  performance  and  SA  in 
UV  control  task 

•  Compare  serial  manual  control  with  parallel  manual  control 
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•  Compare  manual  control  with  automated  control  modes 
-  Investigate  any  relationships  between  response  measures 


Riley,  J.  M.,  &  Endsley,  M.  R.  (2002).  Computer-Aided  Decision  Support:  Is  It  What  the  Army  Needs?  In  In 
Proceedings  of  46th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Baltimore,  MD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

There  has  been  a  call  for  computer-aided  decision  support  in  Army  and  other  military  operations  as  a  result  of  the 
increasing  pace  of  current  and  fbture  warfare.  These  tools  are  expected  to  speed  up  the  critical  thinking  process,  for 
example  in  battle  planning  and  course  of  action  analysis,  by  providing  users  with  critical  information  and  off¬ 
loading  various  cognitive  tasks.  There  is  a  need,  however,  to  determine  the  kinds  of  decision  tools  that  are  best 
suited  to  Army  operational  needs  and  to  consider  the  potential  implementation  issues  associated  with  application  of 
automated  tools  to  complex  operations.  A  structured  approach  is  needed  to  analyze  Army  operations  and  reveal  the 
critical  information  needs  associated  with  the  various  positions,  and  to  determine  what  is  appropriate  in  terms  of 
decision  aiding  systems.  Understanding  information  needs  and  adequately  designing  for  human  integration  with 
decision  tools  will  be  important  to  successfid  overall  system  performance. 


Riley,  J.  M.,  &  Endsley,  M.  R.  (2003).  Supporting  Planning  and  Situation  Awareness  in  Army  Command  and 
Control.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society  Conference. 
Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  planning  process  is  the  initial  basis  for  tactical  maneuvers  and  combat  operations.  Though  plans  are  driven  by 
the  intent  of  the  commanding  officer,  the  quality  of  plans  are  heavily  dependent  upon  the  level  of  situation 
awareness  of  officers  involved  in  the  process.  To  support  Army  planning  in  command  and  control,  it  is  important  for 
us  to  understand  planning  and  the  challenges  to  providing  the  critical  information  underlying  it.  We  conducted  a 
theoretical  investigation  of  maneuvers  planning  in  combat  situations.  We  used  results  from  goal-directed  task 
analysis,  and  insights  from  observations  of  Army  training  exercises  and  experiments,  to  make  inferences  on  the 
activities  carried  out  in  preparation  for  tactical  maneuvers.  Our  goal  was  to  identify  major  characteristics  of  the 
planning  process  in  current  and  future  Army  force  structures.  We  were  also  interested  in  identifying  human  factors 
issues  associated  with  planning  in  a  rapidly  evolving  environment,  and  generating  computer-based  design  concepts 
to  support  situation  awareness  and  decision  making. 


Riley,  J.  M.,  &  Endsley,  M.  R.  (2004).  The  Hunt  for  Situation  Awareness:  Human-Robot  Interaction  in  Search  and 
Rescue.  In  In  Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans, 
LA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Robots  are  gaining  acceptance  as  team  members  in  complex  task  completion,  though  they  must  be  directed  or 
closely  supervised  by  human  operators.  Effective  interaction  between  operators  and  robots  is  dependent  upon  the 
operator’s  ability  to  develop  situation  awareness  on  the  robot  and  environment.  In  teleoperation  this  requires  the 
effective  distribution  of  attention  between  remote  activities  and  the  local  controls  and  interfaces.  Limitations  in 
attentional  resources  can  limit  SA  on  one  or  the  other  place,  resulting  in  overall  decreased  system  performance.  We 
present  observations  made  at  a  robot-assisted  search  and  rescue  exercise  to  describe  common  SA  problems  in 
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vehicle  control.  This  paper  focuses  on  human-robot  interaction  and  the  role  of  SA  in  search  operations.  Key  SA 
concerns  observed  and  discussed  include  difficulties  in  robot  localization,  inadequate  support  for  team  operations 
and  shared  SA,  workload  in  the  visually  demanding  task,  and  poor  integration  of  data  at  the  interface. 


Riley,  J.  M.,  &  Endsley,  M.  R.  (2005).  Situation  Awareness  in  Hri  with  Collaborating  Unmanned  Robotic  Systems. 
In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  future  Army  operations,  soldiers  may  be  required  to  remotely  operate  multiple  robotic  vehicles  and  participate  in 
collaborative  tasks  with  these  systems.  The  ability  to  acquire  and  maintain  situation  awareness  in  tasking  and 
controlling  robots  will  be  critical  to  human-robot  interaction.  Understanding  the  critical  information  requirements 
for  robotics  tasks  will  be  important,  particularly  when  operators  must  work  with  multiple  systems  across  aerial  and 
ground  platforms,  and  must  perform  under  what  will  likely  be  varying  levels  of  system  autonomy.  Here,  we  examine 
SA  needs  in  the  context  of  a  collaborative  military  task  involving  deployment  of  a  single  UAV  that  is  coordinating 
with  multiple  UGVs  to  identify  ‘‘safe  lanes”  for  advancing  troops.  Cognitive  task  analysis  was  conducted  for  the 
task,  along  with  an  examination  of  potential  function  allocations  that  may  require  operator  multi-tasking  and 
frequent  task  switching.  Issues  in  developing  and  maintaining  situation  awareness  are  discussed. 


Riley,  J.  M.,  Endsley,  M.  R.,  Bolstad,  C.  A.,  &  Cuevas,  H.  M.  (2006).  Collaborative  Planning  and  Situation 
Awareness  in  Army  Command  and  Control.  Ergonomics,  49(12-13),  1139-1153. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Command  and  control;  Planning;  Collaboration;  Situation  awareness 
Summary  abstract: 

We  conducted  a  theoretical  investigation  of  a  complex  command  and  control  (C2)  operation — the  manoeuvres 
planning  processes  in  Army  land-battle  situations,  to  improve  understanding  of  how  technology  can  best  be 
designed  to  support  planning  and  course  of  action  development.  We  drew  upon  results  from  cognitive  task  analyses 
and  interviews  with  subject  matter  experts  and  insights  gleaned  from  observations  of  Army  training  exercises  and 
experiments  to  make  inferences  on  the  C2  activities  carried  out  in  preparation  for  tactical  manoeuvres.  In  this  paper, 
we  summarize  several  critical  human  factors  issues  associated  with  planning  in  a  rapidly  evolving  environment,  as 
identified  in  our  investigation,  and  describe  system  design  concepts  aimed  at  addressing  these  challenges  to 
distributed  collaborative  planning  of  C2  activities.  We  conclude  with  implications  for  the  application  of  these 
findings  to  other  C2  domains. 


Riley,  J.  M.,  Endsley,  M.  R.,  Jones,  D.  G.,  &  Bolstad,  C.  A.  (2003).  Integrating  Automation  with  C4isr:  A  User 
Centered  Approach.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and 
Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003  (pp.  175-177).  College  Park,  MD:  U.S. 
Army  Research  Laboratory. 

Key  Words: 

Human  Performance 

Summary  abstract: 

The  desire  for  intelligent  computer  aids  in  Army  and  other  military  operations  has  increased  as  a  result  of  the 
quickening  pace  of  current  warfare  and  the  need  to  dominate  information  to  obtain  tactical  advantage  over  the 


244 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 


opposition  force.  These  tools  are  expected  to  speed  up  the  critical  thinking  process,  for  example  in  battle  planning 
and  course  of  action  analysis,  by  providing  users  with  timely  and  critical  information  and  off-loading  various 
cognitive  tasks.  There  is  a  need,  however,  to  determine  the  kinds  of  decision  aids  and  automated  systems  that  are 
best  suited  to  Army  operational  needs.  A  structured  approach  is  needed  to  analyze  Army  operations  and  soldier 
responsibilities  in  order  to  reveal  the  critical  situation  awareness  requirements  and  to  determine  how  to  support 
decision  making.  Our  initial  research  efforts  have  focused  on  identifying  system  design  features  and  factors 
associated  with  intelligent  aiding  tools  that  might  lead  to  effective  integration  between  the  human  user  and  his/her 
computer  teammate.  A  cognitive  task  analysis  method  was  used  to  identify  situation  awareness  requirements  for 
brigade  level  staff  officers  and  to  develop  position-tailored  decision  displays  to  support  the  decision  making  process. 


Riley,  J.  M.,  &  Kaber,  D.  B.  (2001).  Utility  of  Situation  Awareness  and  Attention  for  Describing  Telepresence 
Experiences  in  a  Virtual  Teloperation  Task.  In  In  Proceedings  of  International  Conference  on  Computer-Aided 
Ergonomics  and  Safety. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  current  study  assessed  the  utility  of  measures  of  situation  awareness  (SA)  and  attention  allocation  for 
objectively  describing  telepresence.  The  concepts  of  SA  and  attention  have  been  identified  as  cognitive  constructs 
potentially  underlying  telepresence  experiences.  The  motivation  for  this  research  was  to  establish  an  objective 
measure  of  telepresence  and  to  further  investigate  the  relationship  between  telepresence  and  teleoperation  task 
performance.  Research  subjects  performed  a  virtual  demining  task  at  varying  levels  of  difficulty  using  a  simulated 
rover  equipped  with  mine  neutralization  tools.  At  the  same  time,  they  completed  two  secondary  monitoring  tasks. 
Teleoperation  task  performance,  SA,  secondary  task  performance  (attention),  and  telepresence  were  measured 
during  the  experiment.  Results  demonstrated  level  of  difficulty  (LOD)  effects  on  performance  and  telepresence. 
Regression  analysis  revealed  LOD  and  attention  to  explain  significant  portions  of  the  variance  in  telepresence. 
Correlation  analyses  revealed  significant  relationships  between  teleoperation  task  performance  and  subjective 
ratings  of  telepresence,  as  well  as  telepresence  and  SA  and  attention. 


Riley,  J.  M.,  Kaber,  D.  B.,  &  Draper,  J.  V.  (2005).  Situation  Awareness  and  Attention  Allocation  Measures  for 
Quantifying  Telepresence  Experiences  in  Teleoperation  Human  Factors  and  Ergonomics  in  Manufacturing,  14(1), 


51-67. 


Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

This  study  assessed  the  utility  of  measures  of  situation  awareness  (SA)  and  attention  allocation  for  quantifying 
telepresence,  the  sense  of  being  present  at  a  remote  site,  in  a  teleoperation  task  scenario.  Attention  and  SA  have  been 
identified  as  cognitive  constructs  potentially  underlying  telepresence.  The  motivation  for  this  research  was  to 
establish  an  objective  measure  of  telepresence  and  investigate  the  relationship  between  telepresence  and 
teleoperation  performance.  Twenty-four  research  participants  performed  a  virtual  ordnance  disposal  task  at  varying 
levels  of  difficulty  (LODs).  The  task  involved  locating,  identifying,  and  disposing  of  virtual  land  mines  in  an 
outdoor  environment  using  a  simulated  remote-control  rover  with  a  robotic  arm.  Performance,  SA,  and  attention 
allocation  were  recorded  along  with  subjective  assessments  of  telepresence.  Results  demonstrated  LOD  effects  on 
performance  and  telepresence.  Regression  analysis  revealed  LOD  and  attention  to  explain  significant  portions  of  the 
variance  in  telepresence.  Results  of  the  study  provide  further  evidence  that  telepresence  may  share  a  relationship 
with  performance,  and  that  cognitive  constructs,  such  as  attention  and  SA,  may  serve  as  alternative,  objective 
measures  of  telepresence.  ©2004  Wiley  Periodicals,  Inc. 
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Riley,  J.  M.,  Murphy,  R.  R.,  &  Endsley,  M.  R.  (2006).  Chapter  25.  Situation  Awareness  in  Control  of  Unmanned 
Ground  Vehicles.  In  N.J.  Cooke,  H.L.  Pringle,  H.K.  Pedersen,  &  O.  Connor  (Eds.),  Human  Factors  of  Remotely 
Piloted  Vehicles.  Series  Title:  Advances  in  Human  Performance  and  Cognitive  Engineering  Research  .  Series 
Editor:  Eduardo  Salas,  Nancy  Cooke,  James  E.  Driskell  &  Anders  Ericsson.  Amsterdam:  Elsevier-  JAI  Press. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Robotic  systems  are  proving  to  be  assets  in  many  civilian  and  military  operations,  particularly  those  that  are  poorly 
suited  to  physical  insertion  of  human  operators  (e.g.,  exploration  of  collapsed  structures  in  which  areas  requiring 
visual  search  are  too  small  and/or  unstable  for  human  entry)  (Murphy,  2004).  In  these  situations,  robotic  entities 
become  the  remotely  controlled  eyes,  ears,  and  sometimes  hands,  of  the  human  operator.  The  robots  are  used  to 
project  the  operator's  presence  and  intent  upon  objects  in  the  remote  space.  The  effectiveness  of  human  interactions 
with  the  robot  system  drives  the  overall  operational  performance.  The  quality  of  this  human-robotic  interaction 
(HRI)  is  highly  dependent  upon  the  operator's  ability  to  develop  and  maintain  situation  awareness  (SA)  during 
performance. 

An  important  issue  in  the  control  of  remotely  operated  vehicles  is  the  limitation  in  development  of  SA  due  to 
impoverished  sensory  information,  attentional  resource  limits,  task/environmental  stressors,  and  system  design 
faults.  In  addition,  where  autonomy  is  added  to  compensate  for  some  of  these  problems  and  provide  for  a  lower 
operator-to-vehicle  ratio,  problems  associated  with  supervisory  control  can  contribute  to  SA  challenges.  This  chapter 
discusses  the  importance  of  SA  and  provision  of  SA  requirements  through  the  interface,  effects  of  task, 
environmental  and  system  factors  on  operator  SA,  and  interface  design  issues  that  impede  SA  development  during 
task  performance.  Examples  of  SA  challenges  are  provided  in  the  context  of  robot-assisted  urban  search  and  rescue 
(USAR)  operations  and  are  derived  from  extensive  experience  in  this  environment  and  naturalistic  observations  of 
remote  control  operations. 


Riley,  J.  M.,  &  Strater,  L.  D.  (2006).  Effects  of  Robot  Control  Mode  on  Situation  Awareness  and  Performance  in  a 
Navigation  Task.  In  In  Proceedings  of  50th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  San 
Francisco  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Our  ability  to  complete  tasks  in  remote  environments  has  improved  with  increased  sophistication  of  robotic  systems. 
The  quality  of  remote  task  performance  is  driven  by  the  quality  of  human  interaction  with  robots  which  can  be 
dependent  upon  the  operator’s  ability  to  acquire  situation  awareness  (SA)  for  task  completion.  There  are,  however, 
often  critical  limitations  in  development  of  SA,  which  are  projected  to  have  a  magnified  effect  when  operators  must 
task  and  control  multiple  unmanned  systems.  In  this  study,  we  investigated  the  effects  the  mode  of  robot  control  on 
operator  performance  and  ability  to  develop  SA  while  navigating  two  robots  through  two  mazes.  Participants  were 
randomly  assigned  to  groups  to  control  the  robots  in  serial  (one  after  another)  or  in  parallel  (at  the  same  time)  either 
manually  or  using  two  types  of  automation.  The  participants  also  viewed  a  systems  interface  to  monitor  the  status  of 
the  robots.  Results  indicate  that  participants  under  serial  control  produced  higher  SA  scores,  though  not  significantly 
different  from  SA  scores  observed  under  parallel  control  modes.  The  best  navigation  performance  was  observed 
under  parallel  control  with  a  high  level  of  autonomy  that  followed  a  prescribed  set  of  navigation  rules.  This  control 
mode  also  resulted  in  the  lowest  workload  scores.  Participants  were  better  able  to  correctly  respond  to  SA  queries 
for  lower  level,  perceptual  items  in  the  robotics  task  as  compared  to  higher  level  comprehension  and  projection 
items.  Performance  in  the  monitoring  task  was  found  to  be  significantly,  positively  correlated  to  participant  SA. 
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Riley,  J.  M.,  Strater,  L.  D.,  Chappell,  S.  L.,  Connors,  E.  S.,  &  Endsley,  M.  R.  (2010).  Chapter  10.  Situation 
Awareness  in  Human-Robot  Interaction:  Challenges  and  User  Interface  Requirements.  In  M.  Barnes  and  F.  Jentsch 
(Eds.),  Human-Robot  Interactions  in  Future  Military  Operations.  London:  Ashgate  Publishing,  Inc. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

In  many  of  today's  complex  military  and  civilian  domains,  remotely  controlled  ground,  aerial,  sea  surface,  and 
underwater  vehicles  are  being  utilized  as  tools  to  extend  the  sensory  and  psychomotor  capabilities  of  the  human  to 
remote  environments.  These  unmanned  robotic  vehicles  facilitate  the  projection  of  the  operators'  presence  and  intent 
upon  the  distant  space.  However,  the  degree  to  which  the  use  of  unmanned  robotic  systems  leads  to  improved  task 
performance  is  dependent  upon  not  only  how  well  the  operator  can  task,  control,  and  manage'  the  robot(s),  but  also 
upon  how  well  the  operator  is  able  to  develop  and  maintain  an  accurate  understanding  of  the  unfolding  situation 
while  interacting  with  the  robotic  system.  Specifically,  operators  need  to  maintain  a  high  level  of  situation 
awareness  (SA)  with  regard  to  both  changes  in  the  environment  and  the  current  and  future  actions  ofthe  robots  under 
their  control.  Indeed,  studies  investigating  issues  surrounding  robot  control  have  pointed  out  that  most  of  the 
problems  encountered  during  robotics  navigation  tasks,  or  other  more  complex  tasks  such  as  search  and  rescue, 
resulted  from  operators'  lack  of  awareness  during  robot  control  (Burke  et  al.,  2004;  Drury  et  al.,  2003).  Difficulties 
such  as  these  limit  the  utility  of  robots  for  enhancing  task  performance  in  remote  environments. 


Riley,  J.  M.,  Strater,  L.  D.,  Davis,  F.,  Strater,  S.,  &  Faulkner,  L.  (2009).  Situation  Awareness  and  Team 
Communications  in  Robot  Control.  In  In  Proceedings  of  53rd  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society,  San  Antonio,  TX. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Operator  performance  in  robotics  tasks  is  impacted  by  the  situation  awareness  (SA)  that  individuals  or  teams  are 
able  to  acquire  and  maintain  on  the  robot,  the  associated  environments,  and  the  tasks  to  be  completed.  SA  can  be 
influenced  by  team  processes  when  multiple  humans  must  work  together  for  coordinating  control  of  multiple  robotic 
systems.  In  this  study,  we  investigated  the  relationship  between  SA  and  team  verbal  communications  in  a 
collaborative  robotics  tasks  involving  robots  under  semi-autonomous  control.  Results  indicate  that  the  content, 
format,  and  amount  of  communication  can  significantly  impact  operator  SA  and  task  performance. 


Riley,  J.  M.,  Strater,  L.  D.,  Sethumadhavan,  A.,  Davis,  F.,  Tharanathan,  A.,  &  Kokini,  C.  (2008).  Performance  and 
SA  Effects  in  Collaborative  Robot  Control  with  Automation.  In  In  Proceedings  of  52nd  Annual  Meeting  of  the 
Human  Factors  and  Ergonomics  Society.  22-26  September  2008.  New  York  City,  NY  USA.  New  York  City,  NY 
USA:  HFES. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Human  operators  of  robotic  systems  need  to  be  able  to  acquire  and  maintain  situation  awareness  (SA)  in  many 
domains  in  order  to  effectively  task  and  control  unmanned  systems.  Overall  task  performance  is  impacted  by  the  SA 
that  individuals  or  teams  develop  on  the  robot,  the  remote  environment,  and  the  tasks  to  be  completed.  Situation 
awareness  can  be  limited  by  attentional  resource  limits,  particularly  when  operators  must  operate  multiple  systems, 
some  of  which  may  be  under  automated  functioning.  In  this  study  we  investigate  the  effects  of  robot  control 
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requirements,  including  robots  under  semi-autonomous  control,  in  a  task  requiring  coordination  and  shared  control 
of  a  robotic  system,  on  task  performance  and  operator  situation  awareness. 


Roesler,  A.,  Feil,  M.,  &  Woods,  D.  D.  (2003).  Design  Is  Sharing  Stories  About  the  Future.  In  In  Proceedings  of 
Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective 
Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research 
Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

In  design  and  development  we  create  and  share  stories  about  the  future  of  operational  settings.  These  stories  concern 
the  impact  of  objects-to-be-created  as  a  source  of  change  in  a  field  of  practice.  The  beneficial  changes  envisioned 
provide  grounds  for  recruiting  investments  and  support  for  prospective  organizational  change.  In  effect  the 
envisioned  impact  justifies  the  efforts  needed  to  create  and  adopt  the  objects  in  question.  However,  research  results 
gathered  after  the  objects  are  introduced  into  fields  of  practice  show  that  the  stories  of  the  future  are  often  quite 
wrong.  In  other  words,  there  is  a  gap  between  the  envisioned  impact  of  a  new  development  and  the  actual  impact 
and  reverberations  of  that  change.  This  paper  analyzes  the  factors  that  make  envisioning  the  future  of  technology 
change  so  fragile  and  uses  this  analysis  to  develop  a  decision  support  method  for  design  as  an  envisioning  process. 


Roesler,  A.,  Feil,  M.,  &  Woods,  D.  D.  (2003).  The  Northwest  Passage  of  Ideation:  How  Does  Design  Find  What 
Would  Be  Useful?  In  In  Proceedings  of  6th  International  Conference  on  Naturalistic  Decision  Making.  Pensacola 
Beach,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Naturalistic  decision  making  began  as  a  way  to  study  practice  in  action.  One  of  the  reasons  to  understand  the  nature 
of  practice  is  to  become  a  participant  in  the  design  of  new  artifacts  intended  to  support  or  expand  expertise.  Thus, 
the  question  arises  how  to  link  research  results  to  the  design  process,  in  particular,  how  does  the  research  process 
and  results  participate  or  influence  the  process  of  defining  what  would  prove  useful  in  the  evolving  field  of  practice. 
To  pursue  the  above  question  creates  a  new  issue  to  resolve  -  what  is  the  nature  of  design  to  improve  cognitive 
work?  The  dexycle  provides  a  general  framework  that  captures  basic  functions  in  research  and  development  as 
design  produces  new  futures  in  an  operational  setting.  Dexycle  represents  the  design  process  as  non-linear, 
distributed  over  multiple  groups  and  with  parallel  cross-connections.  It  is  built  from  observations  of  breakdowns  in 
coordination  and  synchronization  in  design  work  that  result  in  building  the  ‘wrong  system  right.’  Moving  forward  in 
design,  the  dexycle  is  a  synchronization  tool;  moving  back  in  design  it  traces  intent  and  the  basis  for  intent- 
hypotheses  about  what  would  prove  useful. 

Dexycle  is  a  cyclical  structure  based  on  six  design  functions: 

1.  DISTILL:  Field  researchers  study  practice  and  define  valuable  storylines  moving  from  listening  to  stories  of 
practice  (through  many  methods)  to  developing  scenarios  of  success,  failure,  expertise,  and  difficulties. 

2.  IDENTIFY :  Analysts  abstract  patterns  from  the  data  gathered  by  the  field  work  and  establish  a  functional  model 
to  explain  findings.  The  findings  and  concepts  begin  to  define  promising  directions  to  search  for  what  could  prove 
useful.  This  function  moves  from  scenarios  to  concepts  for  what  would  prove  useful. 

3.  SIMULATE:  Through  exploration  design  picks  up  leverage  points  and  cast  design  opportunities.  This  is  a 
discovery  process  which  starts  to  make  promising  directions  into  tangible  functional  prototypes.  In  this  function, 
concepts  become  prototypes. 

4.  DEFINE:  design  now  tunes  the  functional  fidelity  with  considerations  of  feasibility  to  define  realistic 
capabilities  for  realization.  Functional  prototypes  become  detailed  in  order  to  fit  production. 

5.  ADJUST :  Systems  engineering  turns  the  envisioned  artifact  into  a  reproduceable  product  which  can  be  fielded 
(e.g.,  assuring  legal,  safety,  and  market  factors). 
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6.  COLLECT :  Distributors  release  the  product  to  the  public.  This  involves  its  announcement,  introduction  timing, 
and  integration  into  organizations;  the  handling  of  requests  and  revision.  Products  lead  to  new  stories  as  practice 
modifies  artifacts  through  use  and  adapts  to  new  capabilities  and  complexities. 

The  Northwest  passage  refers  to  the  Simulate  function  and  to  the  Identify  processes  leading  into  it  and  the  Defining 
processes  following  from  it.  Interestingly,  the  other  functions  are  marked  by  well  defined  design  artifacts-created  in 
that  process  to  document  and  share  the  activities  and  results.  But  the  Northwest  passage  appears  an  ill  defined  space 
of  opportunities  where  mysterious  processes  of  creativity  operate  but  there  are  great  risks  of  getting  lost  and  failing 
to  find  the  route  to  usefulness. 

New  work  is  trying  the  artifacts  and  products  that  structure  the  Northwest  passage  providing  landmarks  to  guide  a 
creative  but  technically  grounded  process.  Labels  like  affordances,  representation  design  techniques,  reusable  design 
seeds,  animocks  refer  to  these  new  approaches.  To  illustrate  the  difficulties  and  opportunities  of  the  Northwest 
passage,  we  document  a  design  case  that  is  currently  in  mid-passage.  The  design  case  is  to  develop  a  new  concept 
for  control  of  point  of  view  called  ViewT racks  intended  to  help  an  observer  understand  a  scene  s/he  can  access 
through  multiple  virtual  cameras.  As  an  analytical  tool  the  de:cycle  framework  helps  structure  the  design  decisions 
in  play  up  to  this  point  and  as  a  support  tool  it  helps  re-orient  the  design  process  to  overcome  the  tendency  to  get  lost 
in  this  function. 


Roesler,  A.,  Tittle,  J.  S.,  &  Woods,  D.  D.  (2003).  Viewtracks:  The  Importance  of  Point-of-View  in  the  Design  of  3- 
D  Displays.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology 
Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi, 
Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

ViewTracks  is  a  demonstration  of  design  concepts  that  provide  dynamic,  user-controlled  point  of  view  as  a  key  to 
utilize  the  expanding  capabilities  for  3-D  virtual  displays.  Successful  3-D  interfaces  based  on  the  ViewTracks 
concept  will  allow  for  multiple  viewpoints  (or  cameras),  smooth  tracking  across  these  camera  tracks,  and  user 
control  over  camera  location  and  movement  patterns. 


Roesler,  A.,  &  Woods,  D.  D.  (2008).  Chapter  8.  Designing  for  Expertise.  In  H.N.J.  Schifferstein  and  P.  Hekkert 
(Eds.),  Product  Experience.  Amsterdam:  Elsevier. 
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Summary  abstract: 

Designers  are  often  surprised  when  innovative  new  systems  are  introduced  to  the  fi  eld.  Suddenly,  the  new  products 
or  systems  encounter  people,  and  with  them  practices  and  operations  that  have  been  established  before  the  design  of 
a  new  product  was  envisioned.  As  the  new  designs  are  fi  elded,  users  are  confronted  with  the  new  products  and 
wonder  how  they  afford  new  possibilities  and  how  they  can  be  integrated  into  current  activities. 

How  artifacts  are  put  to  use  depends  in  large  part  on  the  perspectives,  initiatives,  and  inquiries  of  these  users.  The 
people  who  will  engage  in  the  application  of  the  new  products  are  in  many  instances  more  than  users;  they  are 
practitioners  of  trained  skills.  They  are  professionals  who  contribute  their  knowledge  to  the  application  of  the  new 
products  and  systems  as  they  utilize  these  in  order  to  accomplish  their  goals.  Practitioners  utilize  products  and 
systems  with  purposes  in  mind.  Do  the  practitioners  consider  a  product  useful?  Can  they  use  it  to  advance  their 
goals?  Does  it  make  sense  to  them  in  terms  of  how  it  infl  uences  or  changes  their  activity? 

As  designers,  we  have  learned  to  study  design  challenges  from  the  perspectives  of  the  people  who  will  put  the  new 
designs  to  use:  How  do  practitioners  know  what  to  do  with  the  new  design?  What  are  the  intentions  and  judgments 
that  guide  operations?  How  do  practitioners  anticipate  what  will  likely  happen  next  as  they  engage  in  a  particular 
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operation  with  a  new  product?  How  can  we  -  as  designers  -  design  artifacts  and  systems  that  support  practitioners, 
so  that  they  excel  in  what  they  are  already  good  at?  How  can  we  integrate  technologies  by  design  to  support  the 
expertise  of  others  or  to  help  create  new  forms  of  expert  activity? 

Understanding  how  practitioners  make  decisions  and  formulate  plans  for  action  is  a  prerequisite  for  designing 
systems  that  facilitate  domain  expertise  in  context.  We  will  illustrate  in  this  chapter  that  expertise  is  comprised  of 
many  factors,  and  relies  on  the  training  and  experience  of  practitioners  in  their  domain.  Expertise  represents  a 
convergence  of  knowledge,  skills,  and  experience  that  results  in  competence  -  the  ability  to  make  appropriate 
decisions  and  assessments  with  regard  to  the  situations  at  hand. 

Experts  operate  in  a  variety  of  domains.  We  fi  nd  them  in  serious  domains  where  they  act  in  high-stakes  functions  as 
surgeons,  pilots,  judges,  commanders,  and  high-level  decision  makers.  Many  of  us  are  experts  in  everyday 
environments,  where  we  converge  special  understanding  and  particular  skills  as  technicians,  designers,  cooks, 
musicians,  wine  connoisseurs,  and  athletes  (Hoffman,  1992). 

Can  just  anyone  who  is  particularly  knowledgeable  in  a  specifi  c  domain  of  human  activity  be  considered  an  expert 
by  others  who  depend  on  the  consequences  of  these  decisions  or  actions?  On  a  day-to-day  basis,  we  use  the  label 
‘expert’  both  as  an  indicator  for  a  high  level  ofprofi  ciency,  training,  and  knowledge  that  is  generally  associated 
with  extended  practical  experience,  and  as  a  social  construct,  where  a  group  of  stakeholders  identify  those  who  are 
more  expert  among  them. 

As  more  radical  innovations  in  the  form  of  technology-intense  systems,  softwaredriven  products,  and  digital  services 
continue  to  transform  our  daily  lives,  being  expert  has  acquired  a  new  meaning.  Those  among  us  who  manage  to 
cope  with  novelty  can  learn  and  adapt  to  be  able  to  catch  up  with  the  continuous  series  of  iterations,  updates  and 
improvements  in  a  continuously  changing  environment.  In  contrast  we  often  fi  nd  lab  researchers  and  developers 
speaking  of  novice  and  expert  users;  in  other  words,  acting  as  if  one  could  divide  the  audience  of  new  technological 
artifacts  into  those  who  are  in  the  know,  and  those  who  do  not  know.  This  has  always  been  a  distortion,  as  expertise 
is  tied  to  a  context  and  depends  on  more  than  just  cumulative  experience. 

What  is  it  that  experts  know  and  how  is  it  relevant  to  design?  Expertise  is  rooted  in  a  particularly  deep-level 
understanding  of  a  domain  of  practice.  This  understanding  utilizes  explanations  and  reasoning  strategies  that  are 
extracted  from  a  conceptual  model  of  the  domain,  constructed  and  continuously  consulted,  tested,  and  updated  by 
the  domain  expert  in  the  course  of  practicing  in  the  domain  by  engaging  in  activities  in  order  to  achieve  goals.  The 
conceptual  model  takes  the  form  of  an  abstract  representation  (Rappaport,  1997;  Heiser  and  Tversky,  2005). 

Briefly,  a  conceptual  model  is  composed  of  functional  and  physical  relationships  in  the  domain  that  were  identifi  ed 
by  the  expert  and  that  allow  him  or  her  to  simulate  what  is  likely  to  happen  next  (Klein  and  Crandall,  1995).  The 
conceptual  model  assists  the  expert  in  fi  nding  explanations  for  observations.  The  expert  has  constructed  this 
conceptual  model  in  the  course  of  exploring  the  domain  along  various  storylines  and  getting  feedback  about  the 
different  events  that  follow  in  various  situations.  Conceptual  models  as  mental  representations  serve  as  structures  for 
reasoning  strategies  that  utilize  actual  observations  in  the  course  of  sense-making  (Klein  et  al.,  2006). 

Design  influences  the  acquisition  of  these  mental  conceptual  models  by  providing  external  representations  that  infl 
uence  how  people  see  the  world  and  obtain  feedback  about  the  world.  Design  infl  uences  the  use  of  these  mental 
conceptual  models  by  changing  the  mental  workload  (memory  and  attentional  loads)  associated  with  making  sense 
of  situations  and  planning  how  to  act.  These  external  representations  mediate  the  user’s  cognitive  work  to  develop 
and  use  conceptual  models  to  make  sense  of  the  world  accessed  through  the  external  devices  and  displays.  Research 
on  expertise  is  concerned  with  issues  that  arise  in  the  course  of  work  with  representations  in  real  settings  -  where 
people  trained  to  different  levels  of  profi  ciency  work  at  tasks  to  different  levels  of  skill.  We  will,  in  the  following, 
focus  on  factors  that  determine  the  knowledge  and  performance  of  experts  -  in  roles  as  operators  of  artifacts  and 
systems,  and  as  stakeholders  that  face  the  complexities  posed  by  novel  technology.  Expert  operators  and  decision¬ 
makers  form  intentions  -  what  needs  to  be  accomplished,  how  to  transform  these  intentions  into  plans  and  actions, 
how  to  recognize  disruptions  to  these  plans,  how  to  cope  with  complexities  and  adapt  their  activities  to  achieve 
goals.  Note  that  we  replace  the  common  terminology  of  ‘user’  with  the  label  practitioner.  We  use  the  latter  term  to 
emphasize  that  people  are  not  passive  recipients  of  products  designed  for  them  or  passive  rule  followers.  Au 
contraire  -  people  actively  make  things  into  tools  and  adapt  plans  to  achieve  goals  as  conditions  and  environments 
change.  Practitioners  will  modify  unsatisfactory  design,  devise  workarounds,  or  simply  abandon  artifacts  that  do  not 
enable  their  purposes. 

Our  objective  as  designers  is  to  support  the  work  of  practitioners  in  context  by  utilizing  innovative  technologies  that 
are  useful,  usable,  and  easily  understood.  Only  if  designers  understand  the  role  and  nature  of  expertise  can  this 
integration  be  possible  and  successful  (Ericsson  et  al.,  2006). 
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Roesler,  A.,  Woods,  D.  D.,  &  Feil,  M.  (2005).  Inventing  the  Future  of  Cognitive  Work:  Navigating  the  'Northwest 
Passage'.  In  W.  Jonas,  R.  Chow,  &  N.  Verhaag  (Eds.),  Design  -System  -  Evolution:  Application  of  Systemic  and 
Evolutionary  Approaches  to  Design  Theory,  Design  Practice,  Design  Research  and  Design  Education.  Also 
Presented  at  the  6th  International  Conference  of  the  European  Academy  of  Design,  University  of  the  Arts.  Bremen, 
Germany.  Bremen,  Germany:  European  Academy  of  Design. 
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Collaborative  Technology  Alliance 
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Cognitive  Systems  Engineering  and  Design,  Design  Methods, 

Human-Centered  Design,  Ideation  Design,  Envisioning,  Broadening, 

Promisingness,  Perspectives  on  Design,  Design  and  Technology  Development, 

Practice-Centered  Design. 

Summary  abstract: 

Computer  scientists,  engineers,  managers,  and  practitioners  make  claims  about  how  new  technologies  will  change 
cognitive  work-how  workers  in  various  fields  of  practice  solve  problems  in  analysis,  fault  management,  control, 
coordination,  and  replanning.  When  systems  are  built  and  fielded  based  on  these  beliefs,  the  actual  effects  on 
practice,  including  new  forms  of  error,  are  quite  different  from  what  was  envisioned  as  users  workaround 
complexities  or  exploit  new  capabilities  (Woods  and  Dekker,  2000).  The  gap  between  hopes  and  reality  in  changing 
the  face  of  cognitive  work  arises  for  2  factors:  (1)  because  claims  for  new  technology  ignore  the  research  findings  of 
the  field  of  Cognitive  Systems  Engineering  on  how  people  cope  with  complexity  and  (2)  because  advocates  for  new 
technology  are  trapped  in  a  narrow  range  of  possible  expressions  of  the  new  capabilities  relative  to  the  demands  of 
cognitive  work.  Since  design  methods  have  not  had  the  desired  impact  of  guiding  designing  in  the  context  of 
cognitive  work,  the  voyage  of  discovery  that  should  follow  from  insight  through  research  has  been  limited.  Concepts 
were  identified,  but  their  implementation  into  the  world  of  practice  calls  for  an  extended  presence  of  design  thinking 
in  technology  application  -  one  that  is  human  centered,  not  technology  oriented  (Winograd  and  Woods,  1997; 
Hoffman  et  al.,  2002;  Hoffman  et  al.,  2004).  This  research  has  examined  how  technologists  envision  the  future  of 
cognitive  work  and  found  a  variety  of  oversimplifications  that  narrow  the  process  of  discovery  (Feltovich  et  al., 
2004).  Based  on  these  results,  the  paper  proposes  an  integration  9fmethods  from  Cognitive  Systems  Engineering  and 
Design  Innovation  for  finding  promising  directions  (e.g.,  Winograd  and  Flores,  1986;  Woods  and  Christoffersen, 
2000;  and  Alexander,  1964;  Jones,  1970  respectively).  The  integration,  or  de:cycle,  coordinates  three  roles  (and 
associated  processes  and  artifacts  produced  through  these  design  processes):  practitioner-how  they  adapt  to 
complexity,  innovator-how  they  envision  what  would  be  useful,  and  technologist-how  they  bring  the  anticipated 
change  into  the  world  of  practice. 


Ross,  K.  G.,  Crandall,  B.,  &  Battaglia,  D.  (2003).  Complex  Cognition  of  Battle  Command  in  the  Objective  Force 
Unit  of  Action  (UA).  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and 
Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University 
College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

Currently,  when  cognition  is  considered  in  the  development  of  systems,  organizations,  and  training,  a  single 
cognitive  element  is  often  studied  in  isolation  and  cognition  is  not  viewed  as  an  inter-connected  flow  during 
performance.  Designs  that  concentrate  on  an  isolated  aspect  of  cognition  may  impede  other  aspects  of  overall 
cognitive  performance.  We  have  developed  a  level  of  analysis,  called  macro-cognition,  for  examining  cognition  in 
field  settings  to  expand  the  assumptions  about  naturalistic  cognition  and  support  more  robust  designs.  In  an 
experiment  conducted  by  the  Battle  Command  Battle  Lab,  Fort  Leavenworth,  KS,  we  applied  techniques  to  access 
and  understand  the  general  flow  of  cognitive  functions  and  processes  as  they  might  occur  in  the  envisioned 
Objective  Force.  This  paper  presents  preliminary  conclusions  about  the  nature  of  macrocognition.  These 
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observations  will  allow  us  to  continue  to  develop  methods  to  understand  the  cognitive  aspects  of  high-quality 
performance  in  the  future  force. 


Ross,  K.  G.,  Klein,  G.,  Thunholm,  P.,  Schmitt,  J.  F.,  &  Baxter,  H.  (2003).  The  Recognitional  Planning  Model: 
Application  for  the  Objective  Force  Unit  of  Action  (UA).  In  In  Proceedings  of  Collaborative  Technology  Alliances 
Conference  2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University 
of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures,  RPM 
Summary  abstract: 

When  the  Objective  Force  is  fielded,  planning  cycles  may  be  significantly  faster.  The  Military  Decision  Making 
Process  (MDMP)  may  run  into  difficulties  with  time  pressure.  The  Army  currently  relies  on  abbreviated  MDMP 
strategies,  but  these  are  not  codified  or  standardized.  A  Recognitional  Planning  Model  (RPM)  has  been  designed  to 
capture  the  actual  planning  process,  as  observed  during  several  field  exercises.  The  RPM  may  have  advantages  for 
Objective  Force  operations,  compared  to  the  MDMP.  An  experiment  was  conducted  at  the  Battle  Command  Battle 
Laboratory  at  Ft.  Leavenworth  to  assess  the  feasibility  of  the  RPM  for  Objective  Force.  A  Unit  of  Action  (UA) 
brigade  staff  was  trained  in  the  RPM  and  then  presented  with  a  series  of  missions.  The  commander  and  staff  had 
little  difficulty  employing  the  RPM,  which  felt  very  natural  to  them.  Some  of  the  participants  were  reluctant  to 
replace  the  MDMP  after  spending  decades  practicing  it,  but  many  others  were  excited  about  a  chance  to  move  into  a 
commander-driven  strategy  for  fast-paced  operations.  It  was  concluded  that  the  RPM  warrants  additional  research  as 
a  potential  planning  approach  for  the  Objective  Force. 


Ross,  M.  G.,  &  Kaelbling,  L.  P.  (2009).  Segmentation  According  to  Natural  Examples:  Learning  Static 
Segmentation  from  Motion  Segmentation.  IEEE  Transactions  on  Pattern  Analysis  and  Machine  Intelligence,  31(4), 
661-676. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Segmentation,  machine  learning,  motion,  computer  vision,  Markov  random  field. 

Summary  abstract: 

Abstract — The  Segmentation  According  to  Natural  Examples  (SANE)  algorithm  learns  to  segment  objects  in  static 
images  from  video  training  data.  SANE  uses  background  subtraction  to  find  the  segmentation  of  moving  objects  in 
videos.  This  provides  object  segmentation  information  for  each  video  frame.  The  collection  of  frames  and 
segmentations  forms  a  training  set  that  SANE  uses  to  learn  the  image  and  shape  properties  of  the  observed  motion 
boundaries.  When  presented  with  new  static  images,  the  trained  model  infers  segmentations  similar  to  the  observed 
motion  segmentations.  SANE  is  a  general  method  for  learning  environment-specific  segmentation  models.  Because 
it  can  automatically  generate  training  data  from  video,  it  can  adapt  to  a  new  environment  and  new  objects  with 
relative  ease,  an  advantage  over  untrained  segmentation  methods  or  those  that  require  human-labeled  training  data. 
By  using  the  local  shape  information  in  the  training  data,  it  outperforms  a  trained  local  boundary  detector.  Its 
performance  is  competitive  with  a  trained  top-down  segmentation  algorithm  that  uses  global  shape.  The  shape 
information  it  learns  from  one  class  of  objects  can  assist  the  segmentation  of  other  classes. 


Rothrock,  L.,  Park,  S.,  Barnes,  M.  J.,  McDermott,  P.,  Hutchins,  S.,  &  Gillan,  D.  (2004).  Systematic  Analysis  of  Risk 
Visualization  Strategies  for  Homeland  Defense.  In  In  Proceedings  of  IEEE  International  Conference  on  Systems, 
Man  and  Cybernetics.  Washington,  DC.  Vol  3,  pp.  2083-2088. 

Key  Words: 
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ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Decision  making,  utility  theory,  analytic 
hierarchy  process,  simulation 

Summary  abstract: 

An  approach  to  conduct  systematic  analysis  of  risk  visualization  strategies  in  a  homeland  missile  defense  context  is 
presented.  The  approach  consists  of  three  phases  which  take  different  perspectives  toward  characterizing 
theproblems  of  resource  management  and  allocation  for  defense  against  missile  attacks.  Decision  makers  in  missile 
defense  are  supervisory  controllers  of  highly  automated  and  resource-constrained  systems.  Our  approach  is 
suggested  as  a  method  to  assess  the  effectiveness  of  visualization  strategies  in  a  dynamic  decision  environment. 

The  three  phases  of  our  approach  included  a  test  for  subject  utilify  preferences,  a  test  for  abject  multiattribute 
preferences  using  the  Analytic  Hierarchy  Process  (AHP).  and  a  test  for  subject  action  selection  using  a  human-in- 
the-loop  simulation  test  bed.  Our  approach  is  proposed  to  systematically  evaluate  the  effectiveness  of  multistage 
decision  making  strategies  against  normative  solutions  to  address  the  problem  of  missile  defense  against  immediate 
and  delayed  threats.  The  alternatives  were  calculated  based  on  expected  cases  of  lives  lost  as  a  result  of  attacks  on 
defended  areas.  Implementation  of  the  test  bed  enabled  us  to  investigate  the  effect  of  visualization  formots  on 
decision  behavior.  Some  preliminary  results  based  on  a  pilot  study  will  be  presented. 

The  implications  of  our  study  extend  beyond  missile  defense.  In  a  worid  where  catastrophic  loss  of  civilian  lives  is 
seen  by  same  as  a  victoy,  those  entrusted  with  homeland  security  must  effectively  manage  limited  resources  to 
combat  terrorist  activities.  Effective  visualization  formats  will  help  to  promote  decisions  that  are  informed  with 
respect  to  the  risks  and  outcomes. 


Saner,  L.  D.,  Bolstad,  C.  A.,  Gonzalez,  C.,  &  Cuevas,  H.  M.  (2009).  Measuring  and  Predicting  Shared  Situation 
Awareness  in  Teams.  In  In  Proceedings  of  3rd  West  Point  Network  Science  Workshop,  West  Point,  NY.  Also  in 
Journal  of  Cognitive  Engineering  and  Decision  Making,  Volume  3,  Number  3,  Fall  2009,  pp.  280-308.  DOI 
10.1518/155534309X474497  ©  2009  Human  Factors  and  Ergonomics  Society. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  order  to  improve  our  understanding  of  situation  awareness  (SA)  in  teams  performing  in  technologically  advanced 
command,  control,  and  communications  (C3)  operations,  researchers  need  to  develop  valid  approaches  to  assess 
both  individual  and  shared  SA.  We  investigated  SA  in  an  interdisciplinary  military  rescue  operation  training 
exercise.  For  this  study,  we  developed  procedures  to  measure  the  degree  of  shared  SA  between  two  team  members 
and  to  improve  the  accuracy  of  their  shared  SA  scores.  We  suggest  that  SA  scores  that  are  calculated  using  many 
existing  methods  may  be  inflated  because  they  often  fail  to  account  for  error  in  terms  of  both  the  amount  of 
information  that  is  thought  to  be  relevant  and  in  the  accuracy  of  a  person’s  knowledge  of  it.  We  calculated  true  SA 
scores  that  account  for  both  of  these  types  of  error.  The  measures  were  then  used  to  evaluate  five  potential  predictors 
of  shared  SA.  Our  analysis  suggested  that  failure  to  compensate  for  error  in  SA  may  lead  to  overestimation  of 
performance  in  a  situation.  The  results  also  revealed  a  significant  relationship  between  shared  SA  and  participants’ 
distance  from  a  central,  joint  service  team,  which  acted  as  the  organizational  hub  within  the  C3  structure.  Shared  SA 
was  better  the  further  away  from  the  hub  people  were,  which  suggests  that  a  person’s  role  and  position  within  an 
organization  affects  the  level  of  shared  SA  that  can  be  achieved  with  other  individuals. 


Saner,  L.  D.,  &  Gonzalez,  C.  (2009).  Naturalistic  Decision  Framing  in  Computer  Mediated  Scientific  Exploration.  In 
In  Proceedings  of  9th  bi-annual  International  Conference  on  Naturalistic  Decision  Making. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Framing,  Decision  Making 

Summary  abstract: 
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This  study  was  conducted  to  determine  how  people  frame  decisions  naturally.  Research  Approach  -  Decision 
framing  was  analyzed  in  the  computer  mediated  Mars  Exploration  Rover  (MER)  mission  planning  process. 
Outcomes  of  proposed  actions  were  coded  as  positively  or  negatively  framed,  with  rates  of  framing  compared 
relative  to  action  specificity  and  expertise.  Findings  -  It  was  found  that  positive  framing  was  preferred  in  general, 
and  that  increased  expertise  led  to  more  balanced  consideration  of  positive  and  negative  outcomes.  Research 
Limitations/Implications  -  This  study  was  exploratory  and  experimentation  is  necessary  to  formalize  the  observed 
patterns  further.  Originality /Value  -  The  findings  suggest  that  providing  decision-makers  additional  information 
about  the  costs  of  using  technology  may  more  rapidly  lead  to  better  mental  representations  in  technology-supported 
decisionmaking.  Take  away  message  -  Different  kinds  of  expert  behavior  promote  different  patterns  of  decision 
framing  relative  to  how  technology  is  used  for  decision-making. 


Santamaria,  A.,  &  Warwick,  W.  (2007).  A  Naturalistic  Approach  to  Adversarial  Behavior:  Modeling  the  Prisoner's 
Dilemma.  In  In  Proceedings  of  Sixteenth  Conference  on  Behavioral  Representations  In  Modeling  and  Simulation 
(BRIMS  '07). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Naturalistic  Decision  Making,  Human  Performance  Modeling,  Prisoner’s  Dilemma 

Summary  abstract: 

In  this  paper  we  present  a  model  of  the  prisoner’s  dilemma  in  which  human-like,  adversarial  behaviors  emerge  from 
interactions  between  two  naturalistic  models.  In  the  prisoner’s  dilemma,  like  other  simple  two-player  games,  the 
behaviors  that  arises  from  iterated  play  can  be  surprisingly  complex.  We  developed  a  model  of  this  task  using  an 
integrated  architecture,  with  a  task  network  representing  the  decision  environment  and  an  underlying  naturalistic 
model  of  decision  making  determining  branching  in  the  task  network.  Several  adaptive  strategies  emerged  in  our 
two-player  models.  In  addition,  we  captured  several  qualitative  effects  of  human  performance. 


Santamaria,  A.,  &  Warwick,  W.  (2008).  Modeling  Probabilistic  Category  Learning  in  a  Task  Network  Model.  In  In 
Proceedings  of  Seventeenth  Conference  on  Behavior  Representation  and  Simulation.  Providence,  RI  SISO. 
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Summary  abstract: 

Task  network  modeling  is  a  powerful  computational  modeling  tool  that  supports  the  analysis  of  complex  systems, 
but  it  is  perceived  to  lack  cognitive  fidelity.  We  describe  an  approach  for  adding  fidelity  to  task  network  models, 
using  a  simple  “naturalistic”  mechanism  based  on  multiple  trace  memory  and  reinforcement-like  learning.  We 
present  a  model  of  the  Weather  Prediction  Task,  a  canonical  example  of  probabilistic  category  learning  and  the  kind 
of  task  that  would  be  difficult  to  represent  in  a  task  network  framework.  This  model  successfully  picked  up  on  the 
statistics  of  its  environment,  differentiating  among  several  probabilistic  outcomes.  This  provides  another  example  in 
our  growing  list  of  human  performance  validations  of  this  modeling  approach. 


Santamaria,  A.,  &  Warwick,  W.  (2009).  Using  a  Naturalistic  Mechanism  to  Capture  Cognition  and  Dynamic 
Behaviors  in  Task  Network  Models:.  In  Proceedings  of  the  18th  Conference  on  Behavior  Representation  in 
Modeling  and  Simulation  (BRIMS),  Sundance,  UT,  31  March  -  2  April  2009. 
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Over  the  past  several  years,  we  have  developed  and  tested  a  computational  implementation  of  recognition-primed 
decision  making  (RPD)  within  Micro  Saint  Sharp  task  network  models.  The  goal  of  this  work  was  to  augment  task 
network  models  to  improve  their  cognitive  fidelity  and  flexibility.  Our  RPD  mechanism  uses  multiple  trace  memory 
with  a  simple  reinforcement-based  learning  to  pick  up  on  the  statistics  of  a  model  environment.  Here  we  present  a 
collection  of  models  and  validation  results  that  include  classic  categorization  and  probabilistic  category  learning 
tasks,  dynamic  adversarial  and  control  tasks,  and  a  larger-scaled  militarily-relevant  task.  The  range  of  tasks  we  have 
modeled  demonstrates  the  flexibility  of  our  approach  and  what  can  be  accomplished  with  simple  mechanisms. 


Sapp,  M.,  &  Gillan,  D.  J.  (2004).  Length  and  Area  Estimation  with  Visual  and  Tactile  Stimuli.  In  In  Proceedings  of 
48th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA.  (pp.  1875-1879). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Why  do  the  psychophysical  functions  for  line  length  (linear)  and  area  (compressive)  differ  and  do  they  differ  for 
both  the  tactile  and  visual  modalities?  Experiments  1 A  and  B  examined  the  effects  of  a  twodimensional  perception 
on  psychophysical  functions  for  visual  perception.  Participants  used  magnitude  estimation  to  judge  the  diameter, 
area,  and  circumference  of  a  set  of  14  circles.  The  psychophysical  functions  for  diameter  was  approximately  1.0,  for 
area  was  approximately  .60,  and  for  circumference  was  above  1.0,  indicating  that  two-dimensional  perception,  per 
se,  does  not  cause  the  compressive  function  for  area.  Obtaining  spatial  information  without  vision  can  be  important 
for  people  with  demanding  graphically  based  decision-making  tasks  and  people  with  visual  impairments.  Tactile 
interfaces  provide  an  alternative  way  to  display  and  obtain  information.  Do  the  tactile  and  visual  modalities  process 
spatial  information  in  similar  ways?  Experiment  2  examined  the  correspondence  between  visual  and  tactile 
perception.  Participants  touched,  but  did  not  see,  a  series  of  circles.  For  each  circle  they  judged  diameter,  area,  and 
circumference.  Psychophysical  functions  for  diameter  length,  circumference  length,  and  area  of  a  circle  estimated  by 
tactile  perception  in  Experiment  2  were  comparable  to  those  for  visual  perception. 


Sapp,  M.  V.,  &  Gillan,  D.  J.  (2003).  A  Perceptual  Paradox:  Line  and  Area  Estimation.  In  In  Proceedings  of 
Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective 
Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research 
Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

Fast  and  accurate  quantitative  judgments,  which  could  aid  decision-making  in  Future  Combat  Systems,  can  be  made 
using  graphical  displays.  Research  has  shown  that  people  accurately  estimate  line  lengths,  but  underestimate  area. 
The  present  research  investigated  possible  psychophysical  and  cognitive  mechanisms  for  the  misestimation  of  area. 
Experiments  1-3  were  motivated  by  Cleveland's  (1983)  observation  that  a  simultaneous  context  —  presenting 
multiple  circles  of  various  sizes  along  with  a  standard  circle  and  the  to-be-judged  target  circle  —  increased  the 
Stevens'  Law  exponent  for  area  from  0.7  to  1.0.  However,  Experiments  1-3  failed  to  replicate  Cleveland's  (1983) 
results.  Experiments  4A  and  4B  examined  possible  processing  differences  involved  in  making  linear  or  area 
judgments.  Experiment  5  looked  at  the  effects  of  dimensionality  on  linear  and  area  judgments.  Finally,  a  hypothesis 
that  estimates  of  circle  areas  involve  a  processing  competition  between  a  linear  component  and  the  area  component 
of  a  circle  was  examined  by  reanalysis  of  area  judgments  from  all  six  experiments  using  a  multiple  regression 
model.  Perceived  Area  =  a  +  bl  (Circumference)  +  b2  (Area).  The  results  showed  a  stronger  relation  for  Perceived 
Area  and  Circumference  in  all  six  experiments.  This  finding  provides  support  for  the  hypothesis  that  the  linear 
aspects  of  a  circle  are  more  salient  than  the  area  aspects,  which  causes  interference  when  making  area  judgments. 


Sarter,  N.,  Ferris,  T.,  &  Hameed,  S.  (2009).  Chapter  9.  Multi-Modal  Information  Exchange  and  Dynamic 
Adaptation.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter: 


255 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 

Foundations  and  Technology.  Section  II.  Presenting  Battlefield  Information  to  Warfighters:  48hrBooks  by  Alion 
Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving 
Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision 
Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board 
Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue 
SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

High  levels  of  complexity  and  dynamism  will  characterize  the  battlefield  of  the  future.  It  will  involve  a  large  number 
of  human  and  machine  agents  in  various  locations  who  need  to  collaborate  on  planning  and  problem-solving  tasks, 
sometimes  under  considerable  time  pressure,  high  risk,  and  uncertainty.  This  implies  that  effective  information 
systems,  i.e.,  the  ability  to  collect,  store,  distribute,  fuse,  and  share  information,  will  become  increasingly  important. 
To  support  this  need,  a  line  of  research  was  conducted  that  focused  on  the  distribution  of  information  across  multiple 
media  and  modalities,  including  vision,  audition,  and  touch.  First,  a  computer-based  simulation  platform  was  created 
that  supports  collocated  and  remote  synchronous  multimodal  communication  and  coordination  among  multiple 
human  and  machine  agents.  The  creation  of  this  rather  unique  platform  was  necessary  to  conduct  controlled  studies 
of  a  wide  range  of  issues  related  to  multimodal  information  processing,  including  modality  preferences,  cross  modal 
interactions,  and  context-sensitive  modality  selection  and  combination.  This  chapter  provides  an  overview  of  the 
main  findings  from  this  line  of  research  and  provides  guidance  for  the  design  of  future  multimodal  displays. 

The  battlefield  of  the  future  will  be  highly  complex  and  dynamic.  It  will  involve  a  large  number  of  human  and 
machine  agents-  some  of  them  collocated,  others  distributed  across  various  locations  -  who  need  to  collaborate  on 
planning  and  problem-solving  tasks,  sometimes  under  considerable  time  pressure,  high  risk,  and  uncertainty.  To 
support  these  tasks,  effective  information  systems  are  needed  that  make  it  easy  to  collect,  store,  distribute.  Rise,  and 
share  information.  With  this  goal  in  mind,  a  line  of  research  was  conducted  that  focused  on  the  distribution  of 
information  across  multiple  media  and  modalities,  including  vision,  audition,  and  touch.  Multimodal  information 
presentation  was  originally  suggested  by  Multiple  Resource  Theory  (MRT)  as  a  promising  means  to  support 
operators  in  coping  with  large  amounts  of  data.  MRT  assumes  that  different  sensory  channels  are  associated  with 
separate  attentional  resources.  Thus,  by  distributing  information  across,  modalities,  resource  competition  should  be 
minimized  and  overall  processing  performance  should  improve.  In  order  to  design  effective  multimodal  displays,  a 
number  of  important  questions  related  to  multimodal  information  processing  were  investigated  as  part  of  this  line  of 
work,  including  (a)  users'  modality  preferences,  (b)  cross  modal  spatial  and  temporal  links  between  vision,  hearing, 
and  touch,  and  (c)  the  design  of  a  context-sensitive  multimodal  interface  that  is  both  adaptive  and  adaptable.  The 
following  sections  will  highlight  some  of  the  major  activities  and  findings  from  this  effort. 


Sarter,  N.  B.  (2002).  Chapter  2.  Multimodal  Information  Presentation  in  Support  of  Human-Automation 
Communication  and  Coordination.  In  E.  Salas  (Ed.),  Advances  in  Human  Performance  and  Cognitive  Engineering 
Research  (Vol.  2,  pp.  13-35).  New  York,  NY:  JAI  Press. 
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Summary  abstract: 

Designers  of  human-machine  interfaces  have  focused  for  a  long  time  on  the  development  and  refinement  of  visual 
and,  to  a  more  limited  extent,  auditory  displays.  Only  in  recent  years  has  a  significant  and  rapidly  increasing  interest 
in  multimodal  systems  emerged.  To  a  large  extent,  this  interest  was  triggered  by  the  evolution  of  virtual  reality  (VR) 
technologies.  Multimodal  interfaces  have  been  developed  that  support  movement  through,  and  interaction  with, 
these  environments  and  thus  increase  the  perceived  immersion  in  virtual  worlds.  Multimodal  technologies  have  been 
proposed  for  other  purposes  and  domains  as  well.  They  involve  the  potential  for  making  computing  accessible  to  a 
wider  range  of  users,  including  people  from  different  age  groups,  with  different  cognitive  styles,  and  those  with 
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sensory  or  motor  impairments.  Multimodal  interfaces  play  an  important  role  in  the  development  of  VR-based 
training  systems  (such  as  surgical  training  systems  for  minimally  invasive  surgery  where  tactile  and  force  feedback 
is  needed),  and  they  are  being  developed  in  support  of  mobile  computing  and  telerobotics. 


Sarter,  N.  B.  (2005).  Graded  and  Multimodal  Interruption  Cueing  in  Support  of  Preattentive  Reference  and  Attention 
Management.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando, 
FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

A  decade  ago,  attention  management  was  considered  by  some  "the  least  explored  frontier  in  cognitive  science  and 
human-machine  cooperation"  (Woods  et  al.,  1994).  Today,  at  least  one  aspect  of  attention  management  - 
interruption  handling  -  still  poses  a  major  challenge  for  the  design  of  human-machine  systems  and  computer- 
supported  collaborative  work.  Successful  interruption  management  requires  that  both  unintentional  dismissals  and 
preemptive  integrations  of  interruption  signals  are  avoided  (Latorella,  1999).  One  means  of  achieving  this  goal  is  to 
support  preattentive  reference,  i.e.,  the  processing  of  interruption  signals  that  occurs  before  attentional  selection. 
Operators  need  to  be  provided  with  at  least  partial  information  about  the  nature  and  cognitive  requirements  of  a 
potential  interruption,  and  this  information  should  be  presented  in  a  way  that  allows  for  peripheral  access.  This  paper 
will  discuss  and  illustrate  how  currently  underutilized  graded  and  multimodal  information  presentation  could  help 
accomplish  this  goal  and  support  various  stages  of  the  overall  interruption  management  process. 


Sarter,  N.  B.  (2006).  Multimodal  Information  Presentation:  Design  Guidance  and  Research  Challenges.  International 
Journal  of  Industrial  Ergonomics  (Special  Issue  on  New  Insights  in  Human  Performance  and  Decision  Making  - 
Invited  contribution),  36(5),  439-445. 
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Summary  abstract: 

Multimodal  interfaces  have  gained  considerable  interest  in  recent  years.  This  trend  can  be  expected  to  continue  as 
designers  try  to  address  challenges  like  data  overload,  the  need  for  improved  performance  of  recognition-based 
systems,  a  greater  sense  of  immersion  in  virtual-reality  environments,  and  support  for  time  sharing  and  attention 
management  in  a  variety  of  complex  real-world  domains.  To  ensure  the  robustness  and  effectiveness  of  multimodal 
interfaces,  a  number  of  design  guidelines  have  been  proposed.  The  present  paper  will  review  guidelines  that  focus 
on  multimodal  information  presentation  (as  opposed  to  multimodal  system  input),  and  it  will  discuss  important 
research  needs  in  this  area,  including  the  exploration  of  possible  uses  and  benefits  of  rarely  used  sensory  channels 
(such  as  touch  and  olfaction),  the  consideration  of  crossmodal  links  in  attention,  the  top-down  modulation  of 
multimodal  information  processing,  and  the  need  for  adaptive  and  adaptable  multimodal  interfaces. 


Sarter,  N.  B.  (2007).  Chapter  13.  Multiple-Resource  Theory  as  a  Basis  for  Multimodal  Interface  Design:  Success 
Stories,  Qualifications,  and  Research  Needs  In  Kramer,  Wiegmann,  &  Kirlik  (Eds.),  Attention:  From  Theory  to 
Practice.:  ADA  CTA. 
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Summary  abstract: 

Multimodal  information  processing  and  design  has  emerged  as  J  maior  research  topic  during  the  past  decade  (e  .g.  , 
Oviatt,  2003;  Sarter,  2002,  2005;  Spence  &  Driver,  J  997,  2004).  This  development  can  be  explained,  in  part,  by  the 
recognition  that  most  naturalistic  situations  involve  simultaneous  111  1  lltimodal  input  (Neisser,  1976).  It  also  reflects 
the  growing  need  for  creating  artificial  multi  modal  environments  and  interfaces  that  effectively  support  diverse 
functions,  such  as  creating  a  sense  of  immersion  in  virtual  reality  environments  or  supporting  mul  titasking  and 
attention  management  in  a  variety  of  complex,  data-rich  domains  (e.g.,  Brickman.  Hettinge  r,  &  Hass,  2000;  Ho, 
Nikolic,  &  Sarter,  2001;  Latorella,  1999;  Means,  Fleischman,  Carpenter,  Szczublewski,  Dingus,  &  Krage,  1993; 
Nikolic  &  Sarter,  200  1;  Sklar  &  Sarter,  1999;  Woods,  1995  ). 

To  date,  the  design  of  most  rnllltimodal  displays  appears  to  have  been  based  -  implicitly  or  explicitlyon  the  original 
version  of  multiple-resource  theory  (MRT)  (e.g.,  Wickens,  1984).  which  postulated  that  people  possess  separate 
fixed-capacity  resources  for  information  processing  that  can  be  characterized  along  three  dimensions:  (1)  the 
processing  stage  1  early  vs.  late  processing),  (2)  the  process  ing  code  (spatial  vs.  verbal  information),  and  (3)  the 
information  modality  (visual  vs.  auditory  encoding;  other  sensory  channels  were  not  considered  in  the  original 
version  of  MRT).  Based  on  MRT,  the  concurrent  performance  of  multiple  ta  sks  should  benefit  to  the  extent  that 
information  related  to  these  tasks  is  presented  in  different  modalities  and  thus  resource  competition  is  reduced. 

Since  its  original  conception,  Wickens  and  colleagues  have  added  qualifications  to  several  aspects  of  MRT  (for  an 
overview,  see  Wickens  [2002]) .  First,  they  emphasized  that  the  multiple-resource  model  was  intended  primarily  to 
predict  the  performance  of  two  or  more  time-shared  continuolls  tasks  (Wickens,  1991  ).  Also,  based  on  a  review  of 
research  methods  and  findings,  Wickens  and  Liu  (1988)  pointed  out  that  benefits  that  have  been  observed  for  cross- 
modal  time-sharing  may  not  necessarily  be  the  result  of  drawing  frO  1 1 1  independent  pools  of  central  perceptual 
resources.  Instead,  peripheral  sensory  factors,  such  as  visual  scanning  or  auditory  masking,  may  play  an  important 
role.  For  example,  studies  that  carefully  controlled  for  visual  scanning  did  not  always  show  performance  benefits  for 
cross-modal  information  presentation.  Most  recently,  the  notion  of  independent  perceptual  resources  has  been 
questioned  based  on  a  considerable  body  of  behavioral  and  neurophysiological  evidence  that  suggests  extensive 
spatial  and  temporal  cross-modal  links  and  constraints  on.  attention,  which  can  both  enhance  and  limit  multi  modal 
information  processing.  (For  an  overview,  see  Spence  and  Driver  [1997,  2004].) 

Although  the  mechanisms  underlying  improved  time-sharing  with  cross-modal  task  and  information  presentation 
continue  to  be  a  matter  of  research  and  debate,  the  phenomenon  itself  has  been  confirmed  and  exploited  in  a  number 
of  research  and  development  efforts.  This  chapter  presents  examples  of  the  successful  implementation  of  multi 
modal  interfaces  in  support  of  concurrent  task  performance  and  information  processing.  It  describes  additional 
benefits  of  distributing  information  across  sensory  channels,  including  redundancy,  complementarity,  and 
substitution.  Finally,  critical  research  needs  in  the  area  of  multi  modal  information  processing  and  interface  design 
are  discussed. 


Sarter,  N.  B.,  Mumaw,  R.  J.,  &  Wickens,  C.  D.  (2007).  Pilots’  Monitoring  Strategies  and  Performance  on 
Automated  Flight  Decks:  An  Empirical  Study  Combining  Behavioral  and  Eye-Tracking  Data  Human  Factors,  49(3), 
347-357. 
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Summary  abstract: 

Objective:  The  objective  of  the  study  was  to  examine  pilots’  automation  monitoring  strategies  and  performance  on 
highly  automated  commercial  flight  decks. 

Background:  Aconsiderable  body  of  research  and  operational  experience  has  documented  breakdowns  in  pilot- 
automation  coordination  on  modern  flight  decks.  These  breakdowns  are  often  considered  symptoms  of  monitoring 
failures  even  though,  to  date,  only  limited  and  mostly  anecdotal  data  exist  concerning  pilots’  monitoring  strategies 
and  performance. 
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Method:  Twenty  experienced  B-747-400  airline  pilots  flew  a  1-hr  scenario  involving  challenging  automation-related 
events  on  a  full-mission  simulator.  Behavioral,  mental  model,  and  eye-tracking  data  were  collected. 

Results:  The  findings  from  this  study  confirm  that  pilots  monitor  basic  flight  parameters  to  a  much  greater  extent 
than  visual  indications  of  the  automation  configuration.  More  specifically,  they  frequently  fail  to  verify  manual 
mode  selections  or  notice  automatic  mode  changes.  In  other  cases,  they  do  not  process  mode  annunciations  in 
sufficient  depth  to  understand  their  implications  for  aircraft  behavior.  Low  system  observability  and  gaps  in 
pilots’understanding  of  complex  automation  modes  were  shown  to  contribute  to  these  problems. 

Conclusion:  Our  findings  describe  and  explain  shortcomings  in  pilot’s  automation  monitoring  strategies  and 
performance  based  on  converging  behavioral,  eyetracking,  and  mental  model  data.  They  confirm  that  monitoring 
failures  are  one  major  contributor  to  breakdowns  in  pilot-automation  interaction. 

Application:  The  findings  from  this  research  can  inform  the  design  of  improved  training  programs  and  automation 
interfaces  that  support  more  effective  system  monitoring. 


Sarter,  N.  B.,  Waters,  M.,  &  Ho,  C.-Y.  (2003).  Supporting  Effective  Communication  and  Coordination  on  the 
Battlefield  through  Adaptive  Multimodal  Information  Exchange.  In  In  Proceedings  of  Collaborative  Technology 
Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003. 
University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
observation 

Summary  abstract: 

The  battlefield  of  the  future  will  involve  a  large  number  of  co-located  and  distributed  agents  who  need  to  collaborate 
on  planning  and  problem-solving  tasks.  They  sometimes  need  to  do  so  under  considerable  time  pressure  and  high 
risk.  This  implies  that  effective  information  systems  and  operations,  i.e.,  the  ability  to  collect,  store,  distribute,  fuse, 
and  share  information,  will  become  increasingly  important  for  the  success  of  U.S.  Army  operations.  One  promising 
way  to  achieve  these  goals  is  the  distribution  of  information  across  sensory  channels.  Multimodal  interfaces  provide 
numerous  benefits.  They  support  an  increase  in  bandwidth  through  concurrent  information  presentation  and 
processing,  redundancy  (i.e.,  several  modalities  can  be  used  for  processing  the  same  information),  synergy  or 
complementarity  (i.e.,  several  modalities  can  be  used  for  processing  different  chunks  of  information  that  need  to  be 
merged),  and  substitution  (i.e.,  one  modality  that  has  become  temporarily  or  permanently  unavailable  is  substituted 
with  some  other  channel).  The  goal  of  our  research  is  to  capitalize  on  these  affordances  in  an  effort  to  design  an 
effective  and  adaptive  multimodal  interface  that  assists  with  distributed  decision-making  and  problem-solving  in 
future  combat  operations.  We  will  identify  and  evaluate  the  effectiveness  of  natural  modality  selection  and 
integration  patterns  for  various  tasks  and  task  contexts,  examine  strategies  for  managing  breakdowns  in  multimodal 
communication,  explore  the  benefits  of  introducing  tactile  cues,  and  address  potential  difficulties  with  the 
concurrent  use  of  more  than  one  modality,  such  as  the  modality  shifting  effect  or  crossmodal  interference. 


SA-Technologies.  (2009).  Chapter  4  -  Communicate.  In  J.  Lipnack,  J.  Stamps,  M.  Prevou,  B.  Veitch,  &  M.  Hannah 
(Eds.),  Teams  of  Leaders  Handbook-  Building  Adaptive,  High-Performing  Interagency  Teams  (Vol.  1).  Fort 
Leavenworth,  Kansas:  Battle  Command  Knowledge  System  (BCKS),  Combined  Arms  Center  -  Knowledge  (CAC- 
K),  Combined  Arms  Center  (CAC).  http://usacac.army.mil/CAC2/bcks/ToL_Handbook.pdf. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 

Interagency  teams,  high  performing  teams,  learning  organization,  leader  competence,  knowledge  management, 
leader  team  development 

Summary  abstract: 

Teams  of  Leaders  (ToL)  is  an  approach  for  rapidly  building  and  effectively  employing  cross-boundary  teams  that 
are  highly  competent  both  in  making  and  executing  decisions  and  in  learning  and  adapting  together.  The  ToL 
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approach  provides  a  deliberate  methodology  for  forming,  launching,  operating,  and  sustaining  nested  leader  teams 
and  developing  their  capacity  to  work  at  higher  performance  levels  in  less  time.  The  ToL  approach  applies  and 
balances  three  key  elements — information  management,  knowledge  management,  and  leader  teams  qualities — to 
boost  team  communication  and  collaboration.  The  value  of  teams  of  leaders  increases  in  proportion  to  the  diversity 
of  team  members,  so  that  the  approach  is  especially  valuable  to  inter-agency  and  coalition  (and  thus  cross-cultural) 
teams. 

This  handbook  provides  a  detailed  explanation  of  both  ToL  theory  and  practice  to  help  doctrine  writers  and 
leadership  development  professionals  integrate  ToL  into  professional  military  education  and  training,  and  it  contains 
worksheets  and  tables  to  help  practitioners  form,  launch,  and  sustain  high  performing  teams. 

Handbook  Objectives  -  What  Does  this  Handbook  do  for  you? 

The  Teams  of  Leaders  Handbook  is  produced  under  the  auspices  of  Combined  Arms  Center-Knowledge  (CAC-K) 
and  its  Battle  Command  Knowledge  System  (BCKS)  component,  this  handbook  lays  out  new  thinking  for  leaders 
working  in  interagency  teams.  Given  the  complexity  of  21st-century  operations,  the  old  top-down,  command-and- 
control  models  of  the  hierarchy  no  longer  suffice.  Instead,  we  need  more  sophisticated  approaches  to  network  the 
force  and  working  across  boundaries  to  improve  collaborating,  team  development,  and  building  relationships,  which 
is  what  this  handbook  sets  out  to  do. 

Our  intention  is  to  refine  and  mature  this  work  with  input  and  reflection  from  those  who  read  and  use  the  ideas 
presented  here.  Together,  we  will  develop  leaders  who  are  situationally  aware,  adaptive,  and  proficient  at  working 
across  boundaries  in  unpredictable  situations  with  peers  whose  backgrounds  may  be  unfamiliar. 


Schoenwald,  J.,  Trent,  S.,  Tittle,  J.,  &  Woods,  D.  (2005).  Scenarios  as  a  Tool  for  Collaborative  Envisioning:  Using 
the  Case  of  New  Sensor  Technologies  for  Military  Urban  Operations.  In  In  Proceedings  of  49th  Annual  Meeting  of 
the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  describes  the  use  of  a  collaborative  envisioning  tool  to  combine  the  goals  of  disparate  communities 
concerning  the  role  of  new  sensor  technologies  being  deployed  in  Military  Operations  in  Urban  Terrain  (MOUT). 
To  do  this,  Cognitive  Engineering  intersects  with  the  sensor  development  and  military  operations  communities. 
Through  the  use  of  scenario-based  design  and  the  Topic  Landscape  tool,  generic  patterns  provide  seeds  that  help 
envision  realistic  futures  of  MOUT  which  are  expressed  in  a  narrative.  These  patterns  provide  insight  on  two  levels: 
On  one  level  they  describe  complexities  inherent  to  all  cognitive  work,  while  on  another  level  they  provide  insight 
about  what  makes  MOUT  difficult.  The  Topic  Landscape  is  a  collaborative  tool  that  organizes  information  from  a 
Cognitive  Task  Analysis  of  MOUT  in  many  forms  (text,  graphics,  video,  etc.)  from  many  contributors.  This 
approach  turns  scenarios  from  a  validation  tool  to  an  effective  envisioning  tool. 


Schraagen,  J.  M.,  Klein,  G.,  &  Hoffman,  R.  R.  (2008).  Chapter  1.  The  Macrocognition  Framework  of  Naturalistic 
Decision  Making.  In  J.M.  Schraagen,  L.G.  Militello,  T.  Ormerod,  &  R.  Lipshitz  (Eds.),  Naturalistic  Decision 
Making  and  Macrocognition  (pp.  3-25).  Aldershot,  England:  Ashgate  Publishing,  Inc. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Naturalistic  Decision  Making  (NDM),  as  a  community  of  practice,  has  the  goal  of  understanding  cognitive  work, 
especially  as  it  is  performed  in  complex  sociotechnical  contexts.  In  recent  years,  the  concept  of  "macrognition"  has 
emerged  as  a  new  and  potential  umbrella  term  to  designate  the  broader  paradigm  that  underlies  NOM.  In  addition. 
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the  notion  of  macrocognition  presents  challenges  and  opportunities  for  both  theory  and  empirical  methodology.  The 
present  volume  is  a  contribution  to  this  literature,  the  seventh  volume  in  the  NDM  series. 

In  this  chapter  we  accomplish  a  number  of  things.  First,  we  chart  the  history  of  NDM  as  a  community  of  practice 
and  then  describe  its  stance  concerning  cognitive  work  and  research  methodology.  Next,  we  chart  the  history  of  the 
concept  of  macrocognition  and  then  show  how  NDM  converges  with  it  philosophically.  Finally,  we  use  these 
contexts  to  overview  the  chapters  in  this  volume. 


Shafto,  M.  G.,  &  Floffman,  R.  R.  (2002).  Editors.  Special  Issue:  Human-Centered  Computing  at  Nasa.  IEEE 
Intelligent  Systems.  IEEE  Intelligent  Systems. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Human-centered  computing,  also  called  “human-centered  systems,” 

•  Focuses  on  creating  new  computational  devices 

•  Is  often  contrasted  with  the  traditional  approach  in  computer  science  that  might  be  dubbed  “machinecentered 
computing” 

That  statement  also  accurately  and  succinctly  describes  NASA’s  working  definition  of  HCC.  In  the  last  issue  of 
Intelligent  Systems,  Robert  Hoffman  and  his  colleagues  presented  a  framework  encompassing  the  various 
approaches  that  constitute  the  world  of  HCC.  This  special  issue  examines  the  particular  region  NASA  is  exploring 
within  this  vast  world —  the  working  definition  of  HCC  shaped  by  NASA’s  mission  requirements,  available 
resources,  and  existing  investments. 


Shattuck,  L.  G.,  &  Miller,  N.  L.  (2004).  A  Process  Tracing  Approach  to  the  Investigation  of  Situated  Cognition.  In 
In  Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Technologists  and  human  factors  practitioners  tend  to  approach  the  measurement  of  situation  awareness  from 
different  perspectives.  Technologists  compare  the  difference  between  the  data  available  in  the  environment  with 
what  has  been  detected  by  the  sensors  built  into  a  system.  Human  factors  practitioners  focus  on  perception  and 
cognition  to  the  exclusion  of  the  technological  parts  of  the  system.  The  authors  propose  a  Dynamic  Model  of 
Situated  Cognition  and  use  it  as  a  framework  for  analyzing  both  the  technological  and  human  aspects  of  a  complex 
system.  They  employ  a  process  tracing  method  in  the  analysis  of  a  high  fidelity  military  command  and  control  (C2) 
simulation.  Their  results  indicate  that  the  model  and  the  process  tracing  method  are  effective  ways  in  which  to 
investigate  the  development  of  situated  cognition  in  complex  systems.  In  addition,  their  results  have  important 
implications  for  designers  of  software,  hardware,  and  training  systems. 


Smith,  M.  W.,  Branlat,  M.,  Stephens,  R.  J.,  &  Woods,  D.  D.  (2008).  Collaboration  Support  Via  Analysis  of  Factions. 
In  In  Proceedings  of  NATO  HFM  Symposium  on  Adaptability  on  Coalition  Teamwork,  Copenhagen,  RTO-MP- 
IST-999. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 
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The  complexity  of  modern  military  operations  causes  challenges  with  the  processes  of  information  analysis  and 
coordination.  The  difficulty  inherent  in  dealing  with  factions  (ubiquitous  in  organizations  and  the  political  realm) 
epitomizes  these  challenges.  Over-simplification  and  neglect  of  distinctions  and  alternatives  are  risks  in  the  analysis 
of  complex  phenomena.  Coordination  with  multiple  agents  is  a  method  for  performing  complex  work  in  general  and 
in  particular  for  mitigating  the  cognitive  challenges  of  analysis  of  complex  phenomena.  However,  lack  of  common 
ground  between  agents  hampers  coordination,  especially  in  unpredictable  and  ambiguous  situations.  We  propose  a 
concept  for  an  analytic  support  tool  based  on  organizing  representations  in  a  framework  designed  to  foster 
exploration,  preventing  individual  analysts  from  overly  narrow  and  reductive  analysis.  As  a  collaborative  tool,  it 
can  serve  as  a  virtual  open  workspace,  mitigating  shortages  in  common  ground  by  allowing  analysts  to  be  informed 
by  one  another’s  explorations  in  the  framework. 


Smith,  M.  W.,  Patterson,  E.  S.,  Zelik,  D.,  &  Woods,  D.  D.  (2007).  Faction  Display:  Visualizing  the  Spectrum  of 
Risk  Estimates  in  a  Terrorist  Attack.  In  In  Proceedings  of  K.Mosier  &  U.  Fischer  (Eds.),  Proceedings  of  the  Eighth 
International  Conference  on  Naturalistic  Decision  Making. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Information  analysis,  visualization,  bias,  intelligence  analysis,  fixation,  representation  aiding 

Summary  abstract: 

Detecting  biased  information,  and  deriving  an  accurate  assessment  of  issues  or  situations  from  a  pool  of  information 
representing  multiple  factions,  are  challenges  for  which  intelligence  analysts  have  developed  strategies.  These 
include  researching  sources,  submitting  products  for  peer  review,  explicitly  contrasting  pro  vs.  con  positions,  and 
comparing  predictions  along  a  spectrum  of  optimism/pessimism.  We  explore  the  concept  of  a  Faction  Display  as  a 
means  to  use  visualization  to  support  awareness  of  the  location  of  a  source  along  a  spectrum  of  opinion.  It  can  be 
used  to  place  estimates  or  other  assessments  in  the  context  of  the  source’s  position  among  the  set  of  factions  in  play. 
The  Faction  Display  concept  is  illustrated  as  a  “design  seed”  placed  within  the  context  of  a  safety  analysis  case 
study;  the  concept  displays  relative  estimates  for  safe  distances  from  Liquefied  Natural  Gas  (LNG)  pool  fires 
resulting  from  a  terrorist  attack. 


Smith,  P.,  Bower,  J.,  &  Spencer,  A.  (2009).  Chapter  23.  Asynchronous  Communication  of  Army  Operations  Orders. 
In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and 
Technology.  Section  IV.  Communicating  Information  across  the  Team:  48hrBooks  by  Alion  Science  & 

Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD 
21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures 
(ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and 
Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue  SE,  Suite 
200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

The  focus  of  this  research  is  on  the  support  of  asynchronous  communication  of  plans  and  plan  modifications  among 
military  units.  The  need  to  better  support  asynchronous  communication  in  the  military  is  evident,  especially  given 
the  increasingly  distributed  battlefield  expected  in  the  U.S.  Army  of  the  future.  In  such  a  distributed  battlefield,  the 
commanders  and  planners  of  operations  will  not  be  colocated  and,  because  of  time  constraints,  will  at  times  require 
asynchr  onous  communication.  This  increased  reliance  on  distributed  responsibility  also  places  more  emphasis  on 
individual  responsibility  to  make  local  decisions,  thus  making  shared  situation  awareness  and  shared  understanding 
of  intent  even  more  important. 

In  the  following,  two  empirical  studies  are  discussed  that  look  at  issues  associated  with  the  communication  of  Army 
operations  orders  and  taskings.  In  the  first  experiment,  46  ROTC  cadets  each  received  a  company  operations  order, 
and  based  on  that  were  required  to  write  a  platoon  operations  order  followed  by  two  recall  questions.  Half  of  the 
cadets  received  the  company  operations  order  in  a  conventional  text  format  that  included  a  static  map  overlay.  The 
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other  half  received  the  same  operations  order,  but  with  4  subsections  presented  with  attached  multi-media 
presentations  that  included  synchronized  animation  and  voice  narration.  The  group  that  viewed  the  multimedia 
presentation  accurately  recalled  mission  critical  information  26%  more  often. 

In  the  second  study,  a  formative  evaluation  was  completed  making  use  of  a  multimedia  communications  tool 
designed  to  support  and  study  asynchronous  communication.  While  participating  in  a  joint  forces  exercise,  ten 
captains,  majors,  and  colonels  (4  soldiers  from  the  United  States,  2  from  Canada,  2  from  France,  1  from  Israel  and  1 
from  Germany)  had  the  option  of  using  this  multimedia  communication  tool  whenever  they  felt  it  would  help  them 
to  communicate  information  to  commanders  in  other  units.  They  created  a  total  of  15  messages.  Two  of  the 
messages  consisted  of  one-way  communications.  The  remaining  13  were  asynchronous  dialogs.  In  these  messages, 
the  officers: 

•  Made  extensive  use  of  pointing,  drawing  and  embedded  written  notes. 

•  Used  these  asynchronous  dialogs  to  detect  and  repair  misconceptions  that  arose  from  live  face-to-face  briefings 
(6/13  dialogs). 

•  Used  these  asynchronous  dialogs  to  share  expertise  while  developing  a  plan  (13/13  dialogs). 


Stanard,  T.,  Lewis,  W.  R.,  Cox,  D.  A.,  Malek,  D.  A.,  Klein,  J.,  &  Matz,  R.  (2004).  An  Exploratory  Qualitative  Study 
of  Computer  Network  Attacker  Cognition.  In  In  Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  New  Orleans.  LA.  (pp.  401-405). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Many  computer  network  defenders  do  not  know  how  malicious  hackers  think  and  act  during  a  network  (McCloskey 
&  Chrenka,  2001).  To  study  attacker  cognition,  experienced  hackers  were  recruited  to  attack  a  Windows  2000 
network  and  pursue  three  goals:  Deface  the  website,  steal  (faux)  credit  card  numbers,  and  read  email.  Participants 
wrote  a  report  of  what  they  did,  and  a  post-attack  cognitive  task  analysis  interview  was  conducted.  Logs  were  also 
captured  on  the  network  including  firewall,  snort  IDS,  and  Microsoft  applications  (IIS,  SQL,  Exchange).  An 
Attacker  Cognition  Model  based  on  data  collected  from  five  participants  was  created.  The  model  has  two  basic 
properties:  It  describes  the  cognitive  steps  followed  by  an  attacker,  and  describes  several  passes  through  these  steps 
that  the  attacker  follows  as  s/he  penetrates  several  layers  deep  into  a  network.  Future  research  using  smaller  sample 
sizes  and  multiple  studies  using  the  same  participants  is  encouraged. 


Stansifer,  C.,  Bennett,  K.  B.,  Talcott,  C.,  Martinez,  S.,  &  Shattuck,  L.  (2001).  Digital  Interfaces  for  Army  Tactical 
Operations.  In  In  Proceedings  of  45th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Minneapolis, 
MN. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Timely  and  accurate  decisions  in  the  Army  domain  are  directly  linked  to  the  commander's  ability  to  assess  the 
battlefield.  The  Army  has  developed  the  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2)  interface  to  be 
used  in  mobile  platforms  to  provide  commanders  with  immediate  access  to  critical  information.  Fig.  I  shows  a 
version  of  this  interface.  The  demonstration  will  illustrate  a  portion  of  the  interface  that  was  simulated  for 
presentation  of  friendly  forces  information. 


Stefanelli,  C.,  Tortonesi,  M.,  Carvalho,  M.,  and  Suri,  N.  (2008).  Network  Conditions  Monitoring  in  the  Mockets 
Communications  Framework.  In  In  Proceedings  of  2007  IEEE  Military  Communications  Conference  (MILCOM 
2007)/  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
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Summary  abstract: 

Communication  between  mobile  devices  in  the  MANET  scenario  exhibits  significant  reliability  and  performance 
problems.  Traditional  communication  infrastructures  designed  for  wired  networks  are  not  well  suited  for  MANETs 
because  they  masquerade  network  level  conditions.  In  the  MANET  scenario,  instead,  there  is  the  need  to  expose 
current  network  conditions  to  applications,  enabling  them  to  adapt  their  behavior  to  changes  in  the  quality  of  the 
communication  links.  In  the  context  of  a  MANET -oriented  communication  framework  (called  Mockets),  this  paper 
presents  a  network  conditions  monitoring  component  which  provides  applications  with  timely  and  accurate 
information  about  communication  channel  characteristics.  In  particular,  the  paper  focuses  on  the  measurement  of 
latency,  obtained  via  Round  Trip  Time  measurement,  for  which  it  presents  three  different  algorithms.  The 
experimental  results  show  the  performances  of  the  different  algorithms  in  a  MANET-like  emulation  environment. 


Stefanelli,  C.,  Tortonesi,  M.,  Benvegnu,  E.,  and  Suri,  N.  (2008).  Session  Mobility  in  the  Mockets  Communication 
Middleware.  In  In  Proceedings  of  IEEE  Symposium  on  Computers  and  Communications  (ISCC  2008),  Marrakech, 
Morocco. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Taking  advantage  of  the  benefits  of  modern  networking,  a  growing  number  of  users  are  exhibiting  mobile  behavior. 
As  they  roam  between  different  network  localities,  they  access  the  Internet  and  the  Web  exploiting  both  wired  and 
wireless  communications  and  using  several  heterogeneous  devices.  Mobile  users  want  to  access  their  subscribed 
services  anywhere,  anytime,  and  want  to  preserve  their  currently  opened  service  sessions  as  they  roam  between 
different  network  localities  or  switch  between  different  devices.  Mobile  users'  requirements  call  for  novel 
middlewares  to  provide  support  for  mobility  on  top  of  the  traditional  Internet  infrastructure.  In  this  context,  we  have 
developed  Mockets,  a  communication  middleware  specifically  designed  to  address  the  challenges  ofwireless 
networks  and  mobile  computing.  In  particular,  Mockets  supports  session  mobility  in  terms  ofseamless  handover  for 
preservation  ofend-toend  connectivity  in  spite  of  node  mobility,  automatic  detection  and  exploitation  of  best 
available  connectivity,  and  migration  of  service  session  endpoints  from  one  node  to  another. 


Strater,  L.,  Scielzo,  S.,  Lenox-Tinsley,  M.,  Bolstad,  C.  A.,  Cuevas,  H.  M.,  &  Endsley,  M.  (2009).  Chapter  20.  Tools 
to  Support  Ad  Hoc  Teams.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the 
Warfighter:  Foundations  and  Technology.  Section  IV.  Communicating  Information  across  the  Team:  48hrBooks 
by  Alion  Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen 
Proving  Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced 
Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management 
Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1100  New  Jersey 
Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

Future  Force  operations  rely  on  rapidly  form,ing  ad  hoc  teams  that  are  mobile,  versatile,  and  distributed  in  time, 
space,  and  purpose,  and  can  bring  to  bear  multiple  perspectives  on  a  problem.  Members  are  convened  for  a  specific 
purpose  and  chosen  primarily  for  their  expertise,  drawing  from  multiple  Army  specialties  and  echelons  as  well  as 
Joint  Forces  or  multinational  entities.  Consequently,  team  members  often  have  different  backgrounds  and 
experiences,  with  few  shared  experiences  at  the  onset  of  their  mission.  The  unique  nature  of  these  teams,  therefore, 
poses  significant  challenges  to  both  military  commanders  and  team  members  who  must  rely  on  one  another  for 
critical  information  to  quickly  develop  sufficient  Team  and  Shared  situation  awareness  (SA)  for  decision  making 
and  achieving  operational  goals. 
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Strater,  L.  D.,  Cuevas,  H.  M.,  Connors,  E.  S.,  Ungvarsky,  D.  M.,  &  Endsley,  M.  R.  (2008).  Situation  Awareness  and 
Collaborative  Tool  Usage  in  Ad  Hoc  Command  and  Control  Teams.  In  In  Proceedings  of  52nd  Annual  Meeting  of 
the  Human  Factors  and  Ergonomics  Society.  22-26  September  2008.  New  York  City,  NY  USA.  New  York  City, 
NY  USA:  HFES. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  research  details  findings  from  Urban  Resolve,  a  large-scale  Joint  Forces  research  effort  to  investigate  the 
impacts  of  technological  solutions  on  situation  awareness  (SA)  and  decision  making  in  ad  hoc  teams  in  a  military 
command  and  control  environment.  The  investigation  was  conducted  using  personnel  acting  as  the  Joint  Force  Land 
Component  Command  (JFLCC)  and  supporting  Brigade-level  staff  stationed  at  Ft.  Leavenworth  in  support  of  the 
higher  headquarters  at  the  Joint  Task  Force  (JTF)  in  an  urban  combat  and  stability  scenario.  In  the  study, 
participants  rated  face-to-face  communication  highest  for  both  routine  and  critical  communications.  The  tool  rated 
most  effective  for  the  development  of  situation  awareness  changed  across  study  sessions,  with  chat  in  chatroom 
rated  highest  in  the  first  and  last  session,  and  shared  map/shared  products  folder  rated  highest  in  the  second  session. 
Results  are  discussed  in  the  context  of  developing  tools  to  support  shared  SA  and  temporal  team  performance  in  ad 
hoc  teams. 


Strater,  L.  D.,  Faulkner,  L.  A.,  Hyatt,  J.  R.,  &  Endsley,  M.  R.  (2006).  Supporting  Situation  Awareness  under  Data 
Overload  in  Command  and  Control  Visualizations.  In  In  Proceedings  of  International  Ergonomics  Association  2006 
Congress,  Maastricht,  Netherlands. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
experimental  study,  Situation  Awareness,  cognition,  human  computer  interaction,  usability.  Command  and  Control 

Summary  abstract: 

Overcoming  information  overload  is  a  central  challenge  for  supporting  information  dense  environments,  such  as 
military  operations,  some  process  control  systems,  and  real-time  data  analysis  tasks.  This  paper  describes  a  study 
which  evaluated  concepts  for  supporting  military  personnel’s  ability  to  quickly  assess  needed  information  in  very 
cluttered  Common  Operating  Picture  (COP)  displays.  The  effort  focused  on  the  presentation  of  four  possible  unit 
representations  on  a  digitized  COP  based  on  ethnographic  observations  of  the  way  in  which  Army  officers  seek  to 
deal  with  the  proliferation  of  large,  overlapping  battlespace  objects  on  dense  displays.  Sixteen  U.S.  Army  Reserve 
Officer  Training  Corps  (ROTC)  cadets  monitored  a  28  minute  simulation  that  depicted  a  developing  military 
advance.  Cadets  viewed  each  of  the  four  modalities  with  presentation  order  counterbalanced  between  subjects  using 
a  Latin  squares  design.  For  each  modality,  cadets  responded  to  a  rapid  battery  of  questions  about  their  SA  using  the 
Situation  Awareness  Global  Assessment  Technique  (SAGAT).  At  the  end  of  the  study,  participants  completed  a 
subjective  questionnaire  about  their  representation  preferences.  While  participants  prefer  current  MilSTD  2525B 
representations,  objective  SA  measures  do  not  support  the  superiority  of  this  representation. 


Strater,  L.  D.,  Riley,  J.  M.,  Faulkner,  L.  A.,  Hyatt,  J.  R.,  &  Endsley,  M.  R.  (2006).  Effect  of  Display  Icon  Modality 
on  Situation  Awareness.  In  In  Proceedings  of  50th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society. 
San  Francisco  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  present  study  investigated  the  effectiveness  of  different  icon  modalities  in  supporting  situation  awareness  (SA) 
when  using  a  densely  populated  battlefield  COP.  The  MilSTD  2525B  symbology  was  compared  to  modified 
versions  of  this  symbology  that  were  either  miniature  or  proportional,  and  the  U.S.  Army  Research  Lab’s  blobology 
concepts.  ROTC  cadets  viewed  a  simulated  military  advance  that  displayed  each  of  the  four  representation 
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modalities  with  presentation  order  counterbalanced.  At  the  end  of  each  segment,  cadets  responded  to  a  rapid  battery 
of  questions  about  their  SA  using  the  Situation  Awareness  Global  Assessment  Technique  (SAGAT).  Participants 
also  evaluated  the  different  icon  modalities  in  terms  of  both  user  preferences  and  the  icons’  perceived  utility  in 
supporting  SA.  Although  participants  perceived  the  current  MilSTD  2525B  representations  as  easiest  for  assisting 
them  in  monitoring  friendly  and  enemy  forces,  the  objective  SAGAT  measure  showed  that  SA  performance  was 
highest  using  the  proportional  icon  modality.  Results  are  discussed  in  the  context  of  designing  unit  representations  to 
effectively  support  SA. 


Strater,  L.  O.,  &  Bolstad,  C.  A.  (2009).  Chapter  7.  Simulation-Based  Situation  Awareness  Training.  In  D.A. 
Vincenzi,  J.A.  Wise,  M.  Mouloua,  &  P.A.  Hancock  (Eds.),  Human  Factors  in  Simulation  and  Training.  Boca  Raton, 
Florida:  CRC  Press,  Taylor  &  Francis  Group. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Simulations  in  SA  Research,  Measurement,  and  Training 
Simulations  in  SA  Research 
Simulations  in  SA  Measurement 
Simulations  in  SA  Training 

Structured  SA-Oriented  Training  Development  Process 

Step  1:  Identify  Domain-Specific  SA  Requirements 

Step  2:  Identify  and  Develop  SA  Measures 

Step  3 :  Identify  SA  Deficits  within  the  Domain 

Step  4:  Develop  Training  Program  to  Target  Selected  Skills 

Step  5:  Validate  Training  Programs 

Step  6:  Identify  Training  Modifications  Needed 

Summary  abstract: 

The  term  situation  awareness  (SA)  has  been  around  since  World  War  II.  Although  originally  associated  with 
military  pilots,  over  the  last  several  years  it  has  emerged  as  a  field  of  research  in  its  own  right  and  is  now  being 
examined  in  other  domains,  such  as  army  operations,  air  traffic  control  (ATC),  nuclear  power  plant  operation,  and 
medical  applications  (Bolstad,  Riley,  Jones,  and  Endsley,  2002;  Collier  and  Folleso,  1995;  Endsley,  1995a;  Gaba, 
Howard,  and  Small,  1995;  Sollenberger  and  Stein,  1995;  Wright,  Taekman,  and  Endsley,  2004).  At  its  core,  SA 
involves  being  aware  of  what  is  happening  around  oneself  to  understand  how  information,  events,  and  one's  own 
actions  will  affect  one's  goals  and  objectives,  both  now  and  in  the  near  future.  In  fact,  SA  is  employed  regularly  in 
the  complex  and  dynamic  tasks  typical  of  everyday  life,  such  as  driving  a  car,  crossing  the  street,  or  juggling  items 
on  a  busy  schedule.  It  is  especially  critical  in  work  domains  where  the  information  flow  can  be  quite  high  and  poor 
decisions  may  lead  to  serious  consequences  (e.g.,  piloting  an  airplane,  functioning  as  a  soldier,  or  treating  critically 
ill  or  injured  patients). 

As  much  of  the  study  of  SA  has  focused  on  those  fields  of  endeavor  where  the  consequences  of  poor  SA  can  be 
quite  severe,  simulations  have  proved  to  be  a  valuable  resource  for  both  basic  and  applied  research  in  these  fields. 
Through  simulations,  one  can  safely  and  objectively  investigate  SA  in  many  complex  and  critical  tasks.  Although 
SA  is  of  great  interest  as  a  basic  research  topic,  its  applicability  to  realistic,  everyday,  yet  critical  tasks  means  that  it 
is  also  of  significant  relevance  in  more  applied  work.  This  application-inspired  perspective  has  initiated  a  process 
that  begins  with  basic  research  on  the  nature  of  SA  within  a  given  field  of  investigation.  Thus,  much  of  the  basic 
research  has  been  driven  by  a  desire  to  improve  SA,  and  subsequently,  performance.  The  research  can  then  be  used 
for  developing  objective  measures  that  can  guide  the  design  and  development  of  equipment,  information  displays,  or 
training  programs  aimed  at  improving  SA. 

Operationally  valid  simulations  provide  an  excellent  vehicle  to  support  these  research  and  design  efforts,  particularly 
for  training  programs  aimed  at  enhancing  SA.  Accordingly,  the  aim  in  this  chapter  is  to  describe  how  simulations 
can  be  employed  throughout  the  process  life  cycle  of  training,  design,  and  evaluation.  We  first  provide  some 
background  information  on  the  SA  construct  and  the  use  of  simulations  in  SA  research,  measurement,  and  training. 
We  then  introduce  our  structured  SA-oriented  training  development  process,  including  a  discussion  on  basic 
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research  and  development  of  measures.  We  illustrate  the  application  of  this  process  with  details  from  one  of  our 
research  projects  aimed  at  developing  simulation-based  SA  training  for  infantry  platoon  leaders. 


Strickland,  J.  L.,  &  Gillan,  D.  J.  (2003).  Static  and  Dynamic  Representations  of  Motion  in  Weather  Displays.  In  In 
Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the 
Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S. 

Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

Visual  displays  of  weather  information  may  provide  valuable  information  for  military  command  and  control.  Critical 
features  of  weather  concern  the  movement  of  the  weather  system  (e.g.,  rate  and  direction  of  movement).  As  a 
consequence  weather  displays  need  to  provide  users  with  motion  information.  The  present  research  compared  two 
static  representations  of  motion  (lines  with  arrowheads  and  plain  lines),  as  well  as  an  animated  display  that  showed 
motion  of  weather  systems  on  maps.  In  Experiment  1,  participants  observed  a  weather  display,  then  answered 
questions  concerning  the  rate  and  direction  of  the  motion,  the  storm  intensity,  etc.  The  accuracy  and  speed  of  their 
responses  were  recorded.  Participants  were  faster  and  more  accurate  when  they  had  observed  the  static  arrow  display 
than  the  static  line  display,  and  were  as  fast  and  accurate  with  the  static  arrow  display  as  with  the  animated  display. 
In  Experiment  2,  participants  observed  an  animated  display  of  the  weather  and  chose  whether  the  static  arrow 
display  or  the  static  line  display  contained  the  same  information.  They  selected  the  static  arrow  display  on  67%  of 
the  trials.  The  data  suggest  that  a  static  arrow  display  provides  motion  information  better  than  a  static  line  display 
and,  in  these  conditions,  at  least,  as  well  as  a  dynamic  display.  The  findings  indicate  that  weather  information 
systems  may  not  require  resource-intensive  dynamic  displays. 


Strickland,  J.  L.,  Gillan,  D.  J.,  &  Chadwick,  R.  (2003).  Static  and  Dynamic  Representations  of  Motion.  In  In 
Proceedings  of  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society,  Conference.  Denver,  CO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Can  static  cues  provide  information  about  motion?  The  present  research  compared  three  types  of  displays  that 
showed  the  motion  of  a  weather  system  -  two  static  displays  (lines  only  and  lines  ending  in  arrows)  and  an  animated 
display.  In  Experiment  1,  participants  received  trials  in  which  all  three  displays  were  presented  and  they  selected  the 
static  display  that  best  provided  the  motion  information  in  the  animated  display.  Participants  selected  the  static 
arrow  display  on  67%  of  the  trials.  In  Experiment  2,  participants  received  trials  with  one  display  at  a  time,  in  which 
they  used  the  display  to  answer  questions  concerning  the  motion  of  the  weather  system.  Performance  with  the  arrow 
(response  time  and  accuracy)  was  better  than  with  the  static  line  and  was  comparable  to  the  animated  display.  These 
results  show  that  static  cues  can  be  interpreted  in  terms  of  motion. 


Suri,  N.,  Carvalho,  M.,  Lott,  J.,  Tortonesi,  M.,  Bradshaw,  J.,  Arguedas,  M.,  and  Breedy  M.  (2007).  Policy-Based 
Bandwidth  Management  for  Tactical  Networks  with  the  Agile  Computing  Middleware.  In  In  Proceedings  of  IEEE 
Military  Communications  Conference  (MilCom  2006)  CD-Rom.  New  York:  IEEE. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Bandwidth  allocation  and  enforcement  in  tactical  networks  is  a  challenging  problem.  The  mobile  ad-hoc  wireless 
environment  is  bandwidth  constrained  and  the  bandwidth  required  by  applications  running  at  any  given  moment  in 
time  typically  exceeds  the  bandwidth  available.  In  addition,  both  network  topology  and  availability  of  network 
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resources  vary  rapidly  in  the  mobile  ad-hoc  scenario.  Therefore,  it  is  important  to  properly  realize  allocation  of 
bandwidth  to  the  competing  applications,  monitoring  of  both  available  and  assigned  network  resources,  and  the 
enforcement  of  constraints  on  channel  usage. 

IHMC’s  agile  computing  middleware  and  KAoS  policy  and  domain  services  components  provide  a  resource 
management  platform  that  enables  dynamic  control  of  application  bandwidth  utilization  in  a  transparent  manner. 
The  agile  computing  middleware  performs  bandwidth  monitoring  and  enforcement.  It  builds  on  top  of  the  Mockets 
library  for  the  realization  of  traditional  client/server  application-level  communications.  The  middleware  also 
integrates  with  the  FlexFeed  component  to  provide  applications  with  publish-subscribe  communications  semantics 
and  to  support  service-specific  instream  data  manipulation. 

The  KAoS  policy  and  domain  services  handle  policy  specification  and  distribution.  KAoS  policies  can  be  used  to 
specify  bandwidth  limits  based  on  hosts,  port  numbers,  and/or  data  flows. 


Suri,  N.,  Rebeschini,  M.,  Breedy,  M.,  Carvalho,  M.,  and  Arguedas,  M.  (2007).  Resource  and  Service  Discovery  in 
Wireless  Ad-Hoc  Networks  with  Agile  Computing.  In  In  Proceedings  of  IEEE  Military  Communications 
Conference  (MilCom  2006)  (CD-Rom.  New  York:  IEEE. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Resource  and  service  discovery  in  tactical  networking  environments  is  a  challenging  problem.  The  ad-hoc  and  peer- 
to-peer  nature  of  the  network  invalidates  many  traditional  approaches  to  resource  and  service  discovery  that  rely  on 
registries.  The  agile  computing  middleware  supports  opportunistic  service  and  resource  discovery  and  tasking  in 
tactical  environments.  The  group  manager  component  of  the  middleware  supports  bandwidth-efficient  discovery  of 
neighboring  nodes,  their  resources  (including  CPU,  memory,  storage,  and  connectivity),  and  their  services.  Nodes 
may  join  one  or  more  groups  that  allow  resources  and  services  to  be  categorized  into  related  sets.  Applications  using 
the  group  manager  are  notified  about  the  appearance  and  disappearance  of  peers  as  well  as  changes  in  memberships 
of  groups.  The  group  manager  is  efficient  in  finding  neighboring  services  and  is  designed  to  improve  the  probability 
of  finding  resource-rich  nodes.  Crosslayer  integration  between  the  group  manager  and  the  MANET  layer  reduces  the 
network  overhead  and  improves  performance. 


Suri,  N.  (2009).  Agile  Computing  and  Adaptive  Systems.  In  Invited  seminar  at  the  University  of  Modena  and 
Reggio  Emilia,  Modena,  Italy.  June  4,  2009. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

•  Agile  Computing 

-  Agile  Computing  Overview 

-  Mockets  Communications  Library 

-  Group  Manager  Discovery  service 

-  FlexFeed  Publish-Subscribe  System 

-  AgServe  Service-oriented  Architecture 

-  DisService  Information  Dissemination  System 

•  Process  Integrated  Mechanisms 

•  Federated  Information  Spaces 

•  Work  in  Progress  /  Future  Ideas 

-  DAIMS  -  Disaster  Area  Information  Management  System 

-  Information  Oriented  Networking 

-  Network  Science 

Summary  abstract: 

Agile  Computing  Definition 

•  Both  a  Metaphor  and  an  Approach  to  Distributed  Information  Systems 

•  Design  Systems  to  be 
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-  Opportunistic  in  discovering,  manipulating,  and  exploiting  computing  and  communication  resources 

-  Quick  in  reacting  to  changes  in  the  environment 

-  Able  to  take  advantage  of  the  “wiggle  room”  available  in  the  system 


Suri,  N.,  Benvegnu,  E.,  Tortonesi,  M.,  Stefanelli,  C.,  Kovach,  J.,  &  Hanna,  J.  (2009).  Communications  Middleware 
for  Tactical  Environments:  Observations,  Experiences,  and  Lessons  Learned.  IEEE  Communications  Magazine, 
47(10),  56-63. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Tactical  networking  environments  present  significant  challenges  that  must  be  overcome  in  order  to  effectively 
support  net-centric  warfare.  The  wireless  and  ad  hoc  nature  of  these  networks  implies  unreliable  connectivity, 
limited  bandwidth,  and  variable  latency.  Past  and  current  research  has  focused  on  physical  and  data  link  layers, 
routing  protocols,  transport  protocols,  and  cross-layer  aspects.  However,  significant  work  is  needed  at  the  upper 
layers  to  better  support  application  requirements.  In  our  experience,  achieving  effective  communications  in  tactical 
environments  requires  taking  into  account  application  requirements  and  communication  patterns,  designing  a  rich 
interface  between  the  application  and  communication  layers,  and  realizing  a  communications  middleware 
specifically  adapted  to  tactical  networks.  In  this  article,  we  report  on  our  observations  from  several  tactical 
networking  experiments  and  demonstrations  and  the  lessons  learned  from  deployment  of  the  Mockets  middleware  to 
support  tactical  communications.  We  hope  these  experiences  are  useful  to  others  designing  and  implementing 
applications  and  systems  for  tactical  environments. 


Suri,  N.,  Bradshaw,  J.,  Burstein,  M.,  Uszok,  A.,  Benyo,  B.,  Breedy,  M.,  Carvalho,  M.,  Diller,  D.,  Groth,  P.,  Jeffers, 
R.,  Johnson,  M.,  Kulkami,  S.,  &  Lott,  J.  (2003).  Toward  DAML-Based  Policy  Enforcement  for  Semantic  Data 
Transformation  and  Filtering  in  Multi-Agent  Systems.  In  In  Proceedings  of  2nd  International  Joint  Conference  on 
Autonomous  Agents  and  Multi-Agent  Systems  (AAMAS  2003).  Melbourne,  Australia. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Software  agents  have  been  proposed  as  a  solution  to  integrate  existing  heterogeneous  and  stovepiped  information 
systems.  These  capabilities  extend  agent  autonomy  to  encompass  communication  and  information  exchange. 
However,  in  sensitive  or  critical  application  areas  such  as  the  military  or  competitive  business  environments 
involving  proprietary  information  this  kind  of  transparent  interoperability  raises  concerns  about  the  nature  of  the 
information  being  exchanged. 

This  paper  describes  an  approach  to  provide  runtime  policy-based  control  over  information  exchange.  Two  different 
control  mechanisms  are  discussed:  semantic  (content-based)  filtering  of  messages  as  well  as  in-stream 
transformation  of  messages.  Both  of  these  control  mechanisms  are  driven  by  policies  at  run-time.  These  mechanisms 
allow  a  far  more  fine-grained  control  over  dynamic  and  autonomous  agent  interactions.  With  such  an  approach,  we 
hope  to  increase  the  confidence  with  which  system  designers  will  adopt  agent-based  approaches  to  building 
dynamic,  heterogeneous  systems. 

A  companion  paper  [1]  provides  more  details  about  the  underlying  KAoS  policy  and  domain  services  and  the 
NOMADS  mobile  agent  system  as  well  as  the  DAML-based  representation  of  policies.  The  Coalition  Agents 
Experiment  (CoAX)  which  provided  the  testbed  for  this  work  is  described  in  [2], 


Suri,  N.,  Bradshaw,  J.  M.,  Burstein,  M.  H.,  Uszok,  A.,  Benyo,  B.,  Breedy,  M.  R.,  Carvalho,  M.,  Diller,  D.,  Groth,  P. 
T.,  Jeffers,  R.,  Johnson,  M.,  Kulkami,  S.,  &  Lott,  J.  (2003).  DAML-Based  Policy  Enforcement  for  Semantic  Data 
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Transformation  and  Filtering  in  Multi-Agent  Systems.  In  In  Proceedings  of  Autonomous  Agents  and  Multi-Agent 
Systems  Conference  (AAMAS  2003).  Melbourne,  Australia.  New  York,  NY :  ACM  Press,  pp.  1132-1133.  Also  in 
Proceedings  of  Third  International  Central  and  Eastern  European  Conference  on  Multi-Agent  Systems  (CEEMAS 
2003).  Prague,  Czech  Republic. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Policies,  Policy-based  Control,  Information  Release  Policies, 

Semantic  Filtering,  Data  Transformation  Policies,  Coalition 
Operations,  CoAX,  CoABS  Grid,  DAML,  DAML-S,  KAoS, 

Nomads. 

Summary  abstract: 

This  paper  describes  an  approach  to  runtime  policy-based  control  over  information  exchange  that  allows  a  far  more 
fine-grained  control  of  these  dynamically  discovered  agent  interactions.  The  DARPA  Agent  Markup  Language 
(DAML)  is  used  to  represent  policies  that  may  either  filter  messages  based  on  their  semantic  content  or  transform 
the  messages  to  make  them  suitable  to  be  released.  Policy  definition,  management,  and  enforcement  are  realized  as 
part  of  the  KAoS  architecture.  The  solutions  presented  have  been  tested  in  the  Coalition  Agents  Experiment  (CoAX) 
-  an  experiment  involving  coalition  military  operations. 


Suri,  N.,  Bradshaw,  J.  M.,  Carvalho,  M.,  Breedy,  M.,  Cowin,  t.,  Saavedra,  R.,  &  Kulkarni,  S.  (2003).  Applying 
Agile  Computing  to  Support  Efficient  and  Policy-Controlled  Sensor  Information  Feeds  in  the  Army  Future  Combat 
Systems  (FCS)  Environment.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science 
and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003  (pp.  191-196).  College  Park,  MD:  U.S. 
Army  Research  Laboratory. 

Key  Words: 

Human  Performance 

Summary  abstract: 

Agile  computing  is  an  approach  to  opportunistically  discover  and  exploit  available  computing  and  communication 
resources  to  improve  peiformance,  faulttolerance,  efficiency,  and  survivability.  This  paper  describes  the  FlexFeed 
architecture  that  realizes  the  notion  of  agile  computing  in  the  context  of  the  Army  Future  Combat  Systems.  FlexFeed 
provides  efficient  sensor  information  feeds  while  at  the  same  time  enforcing  required  policy  constraints.  FlexFeed 
tries  to  minimize  network  bandwidth  utilization,  distribute  processing  load  across  multiple  nodes,  and  satisfy 
requirements  on  utilization  of  power-constrained  devices. 


Suri,  N.,  Carvalho,  M.,  Ansaloni,  D.,  Arguedas,  M.,  Benincasa,  G.,  Bevegnu,  E.,  Bradshaw,  J.  M.,  Breedy,  M., 

Choy,  S.,  Kovach,  J.,  Marconi,  M.,  Quitadamo,  R.,  Tokarcik,  M.  R.,  Tortonesi,  M.,  &  Winkler,  R.  (2009).  Chapter  3. 
Agile  Computing  and  Its  Applications  to  Tactical  Military  Environments.  In  P.  McDermott  and  L.  Allender  (Eds.), 
Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  I.  Collecting, 
Processing  and  Distributing  Battlefield  Information:  48hrBooks  by  Alion  Science  &  Technology  Corporation  under 
contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and 
Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative 
Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the 
Soldier  Decision  Maker.  July  22-23,  2009.  1100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003. 
http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 
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Agile  computing  is  an  innovative  metaphor  for  distributed  computing  systems,  which  prescribes  a  new  approach  to 
their  design  and  implementation.  This  chapter  describes  the  overall  agile  computing  metaphor  as  well  as  one 
concrete  realization  through  a  middleware  infrastructure. 

Agile  computing  may  be  defined  as  the  opportunistic  discovery,  manipulation,  and  exploitation  of  available 
computing  and  communication  resources  in  order  to  improve  capability,  performance,  efficiency,  fault-tolerance, 
and  survivability.  The  term  agile  is  used  to  highlight  the  desire  to  both  quickly  react  to  changes  in  the  environment 
as  well  as  to  take  advantage  of  transient  resources  only  available  for  short  periods  of  time.  Agile  computing  thrives 
in  the  presence  of  highly  dynamic  environments  and  resources,  where  nodes  are  constantly  being  added,  removed, 
and  moved,  resulting  in  intermittent  availability  of  resources  and  changes  in  network  reachability,  bandwidth,  and 
latency. 

From  a  high-level  perspective,  the  goal  of  agile  computing  is  to  facilitate  resource  sharing  among  distributed 
computing  systems.  At  this  broad  level  of  description,  agile  computing  overlaps  with  several  other  areas  of  research 
including  distributed  processing,  peer-to-peer  resource  sharing,  grid  computing,  and  cluster  computing. 


Suri,  N.,  Marcon,  M.,  Quitadamo,  R.,  Rebeschini,  M„  Arguedas,  M.,  Stabellini,  S.,  Tortonesi,  M.,  &  Stefanelli,  C. 
(2008).  An  Adaptive  and  Efficient  Peer-to-Peer  Service-Oriented  Architecture  for  Manet  Environments  with  Agile 
Computing.  In  In  Proceedings  of  Second  IEEE  Workshop  on  Autonomic  Computing  and  Network  Management 
(ACNM'08)  Brazil. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Agile  Computing,  Peer-to-Peer,  Service,  Discovery,  Service  Migration,  Service-oriented;  Architectures,  JXTA. 

Summary  abstract: 

Realizing  adaptive  and  efficient  peer-to-peer  Service-oriented  Architectures  for  MANET  environments  is  a 
challenging  problem.  In  particular,  robust  and  efficient  service  discovery  and  service  migration  are  critical  in  the 
constantly  changing  and  bandwidth  limited  MANET  environments.  In  these  scenarios,  service  migration  lays  the 
foundation  for  self-adaptive  architectures.  This  paper  describes  the  agile  computing  approach  to  peer-to-peer  service 
discovery  and  service  migration  and  provides  a  performance  evaluation  of  these  functions  in  the  context  of  the  Agile 
Computing  middleware.  The  experimental  results  presented  in  the  paper  show  that  applications  built  on  top  of  the 
Agile  Computing  middleware  are  capable  of  opportunistically  exploiting  transient  computing  resources  in  the 
MANET  environment. 


Suri,  N.,  Rebeschini,  M.,  Arguedas,  M.,  Carvalho,  M.,  Stabellini,  S.,  &  Breedy,  M.  (2007).  Towards  an  Agile 
Computing  Approach  to  Dynamic  and  Adaptive  Service-Oriented  Architectures.  In  In  Proceedings  of  First  IEEE 
Workshop  on  Autonomic  Communication  and  Network  Management  (ACNM'07).  New  York:  IEEE. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Terms-Agile  Computing,  Service-oriented  Architectures,  Dynamic  and  Adaptive  Middleware 

Summary  abstract: 

Agile  computing  is  an  innovative  metaphor  for  distributed  computing  systems  and  prescribes  a  new  approach  to 
their  design  and  implementation.  Agile  computing  may  be  defined  as  opportunistically  discovering,  manipulating, 
and  exploiting  available  computing  and  communication  resources  in  order  to  improve  capability,  performance, 
efficiency,  faulttolerance,  and  survivability. 

This  paper  describes  the  realization  of  the  middleware  and  the  AgServe  library  that  supports  dynamic  and  adaptive 
serviceoriented  architectures.  The  middleware  supports  service  definition,  instantiation,  invocation,  relocation,  and 
termination  to  be  performed  dynamically  at  runtime.  A  coordination  mechanism  continuously  monitors  service 
resource  utilization,  invocation  patterns,  and  network  and  node  resource  availability  to  determine  optimal  locations 
for  services  to  be  instantiated  and  invoked.  The  continuously  adaptive  nature  of  agile  computing  makes  it  well  suited 
to  providing  a  foundation  for  autonomic  computing. 
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Suri,  N.,  Tortonesi,  M.,  Arguedas,  M.,  Breedy,  M.,  Carvalho,  M.,  &  Winkler,  R.  (2006).  Mockets:  A  Comprehensive 
Application-Level  Communications  Library.  In  In  Proceedings  of  MILCOM  2005  Conference,  Atlantic  City,  NJ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Mockets  is  a  comprehensive  communications  library  designed  to  address  challenges  specific  to  mobile  ad-hoc 
networks.  Mockets  have  been  implemented  at  the  application-level  to  simplify  deployment  and  portability.  Both 
stream-oriented  and  message-oriented  abstractions  are  supported,  with  the  message-oriented  service  providing 
multiple  classes  of  service  (reliable,  unreliable,  sequenced,  unsequenced),  message  tagging  and  replacement,  and 
prioritization.  Mockets  also  interfaces  with  a  policy  management  infrastructure  to  support  bandwidth  limitation. 
Finally,  mockets  supports  transparent  migration  of  communication  endpoints  across  hosts  without  the  need  to 
terminate  and  reestablish  connections.  Mockets  provides  similar  semantics  to  TCP  but  performs  better  than  TCP  on 
adhoc  networks. 


Talcott,  C.,  Bennett,  K.  B.,  Martinez,  S.,  Stansifer,  C.,  &  Shattuck,  L.  (2001).  An  Empirical  Evaluation  of  Interfaces 
for  Army  Tactical  Operations.  In  In  Proceedings  of  45th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society.  Minneapolis,  MN. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Timely  and  accurate  decisions  in  the  Army  domain  are  directly  linked  to  the  commander's  ability  to  assess  the 
battlefield.  We  reviewed  both  existing  and  proposed  Army  interfaces  and  applied  a  Cognitive  Systems  Engineering 
(CSE)  approach  to  the  development  of  an  alternative  interface:  the  RAPTOR  interface  (Representation  Aiding 
Portrayal  of  Tactical  Operations  Resources).  This  interface  and  a  version  of  the  Army's  recently-implemented  digital 
interface  (the  Force  XXI  Battle  Command  Brigade  and  Below  —  FBCB2)  were  evaluated  using  a  simulated 
offensive  scenario  and  domain  experts  from  an  active  Army  unit.  The  results  indicate  that  the  RAPTOR  interface 
produced  significantly  better  performance  for  quantitative,  categorical  and  needs  assessments.  It  is  concluded  that 
the  RAPTOR  interface  summarizes  critical  information  for  Army  units  at  all  levels  and  makes  this  information 
easily  accessible  through  effective  graphical  formats  and  interaction  style. 


Talcott,  C.  P.,  Bennett,  K.  B.,  Martinez,  S.  G.,  Shattuck,  L.  G.,  &  Stansifer,  C.  (2008).  Perception-Action  Icons:  An 
Interface  Design  Strategy  for  Intermediate  Domains.  Human  Factors,  49(1),  120-135. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

Objective:  Aprototype  interface  was  developed  to  support  decision  making  during  tactical  operations;  a  laboratory 
experiment  was  conducted  to  evaluate  the  capability  of  this  interface  to  support  a  critical  activity  (i.e.,  obtaining  the 
status  of  friendly  combat  resources). 

Background:  Effective  interface  design  strategies  have  been  developed  for  domains  that  have  primarily  law-driven 
(e.g.,  process  control)  or  intentdriven  (e.g.,  information  retrieval)  constraints.  However,  design  strategies  for 
intermediate  domains  in  which  both  types  of  constraints  are  equally  critical,  such  as  military  command  and  control, 
have  not  been  explored  as  extensively.  The  principles  of  direct  perception,  direct  manipulation,  and  perception- 
action  loops  were  used  to  develop  a  hybrid  interface  design  strategy  (“perception-action  icons”)  that  was 
incorporated  into  the  prototype  interface. 
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Methods:Aqualitative  tactical  simulation  and  an  alternative  interface  (an  experimental  version  of  an  existing  U.S. 
Army  interface)  were  developed.  Participants  used  both  interfaces  to  provide  estimates  of  friendly  combat  resources 
for  three  different  categories  of  information  at  three  different  echelon  levels. 

Results:  The  results  were  unequivocal,  indicating  that  the  interface  with  perception-action  icons  produced 
significantly  better  performance. 

Conclusion:  The  perception-action  icon  design  strategy  was  very  effective  in  this  experimental  context.  The 
potential  for  this  design  strategy  to  be  useful  for  other  intermediate  domains  is  explored. 

Application:  Actual  or  potential  applications  of  this  research  include  both  specific  interface  design  strategies  for 
military  command  and  control  and  general  interface  design  principles  for  intermediate  work  domains. 


Tate,  A.,  Dalton,  J.,  Bradshaw,  J.  M.,  &  Uszok,  A.  (2005).  Agent  Systems  for  Coalition  Search  and  Rescue  Task 
Support.  In  In  Proceedings  of  Third  Workshop  on  Knowledge  Systems  for  Coalition  Operations  (KSCO  2004). 
Pensacola,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  Coalition  Search  and  Rescue  Task  Support  project  shows  cooperative  agents  supporting  a  highly  dynamic 
mission  in  which  AI  task  planning,  inter-agent  collaboration,  workflow  enactment,  policy-managed  services, 
semantic  web  queries,  semantic  web  services  matchmaking  and  knowledge-based  notifications  are  employed. 


Tate,  A.,  Dalton,  J.,  Siebra,  C.  d.,  Aitken,  S.,  Bradshaw,  J.  M.,  &  Uszok,  A.  (2004).  Intelligent  Agents  for  Coalition 
Search  and  Rescue  Task  Support.  In  In  Proceedings  of  Nineteenth  National  Conference  on  Artificial  Intelligence 
(AAAI-2004).  San  Jose,  CA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  Coalition  Search  and  Rescue  Task  Support  demonstration  shows  cooperative  agents  supporting  a  highly 
dynamic  mission  in  which  AI  task  planning,  inter-agent  collaboration,  workflow  enactment,  policy-managed 
communications,  semantic  web  queries,  semantic  web  services  matchmaking  and  knowledge-based  notifications  are 
employed. 


Tate,  A.,  Dalton,  J.,  Uszok,  A.,  &  Bradshaw,  J.  (2004).  Ai  Planning  for  Grid/Web  Services  Composition,  Policy 
Analysis  and  Workflow.  In  In  Proceedings  of  DARPA  DAML  PI  meeting.  New  York,  NY. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

AIAI  Summary  Report 

2003  Goal 

-  Link  I-X  coordination  and  task  support  with  KAoS  agent,  domain  and  policy  services 

-  Demonstrate  in  a  Search  &  Rescue  scenario  in  TTCP  Binni  C2  Domain 

-  To  be  shown  as  AAAI-2004  Intelligent  Systems  Demonstrator 
http://www.aiai.ed.ac.uk/project/cosar-ts/demo/isd/ 

2004  Goal 

-  Create  a  web  service  composition  tool  based  on  AI  planning  technology  that  can  account  for  execution  policy 
issues,  requirements  and  constraints 

Release  Plans 

-  Currently  I-X  version  3.3  and  CoSAR  demonstration  are  available  via  web  for  research  use 
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-  Open  source  I-X  version  4.0  for  research  and  US  government  use  planned  for  September  2004.  Tool  based  on  this 
put  on  SemWebCentral  soon  after. 

Plans  to  end  of  Project 

-  Do  our  best  to  package  the  results  (effort  mostly  used  to  date) 

-  Do  our  best  to  continue  to  participate  in  SWSL  and  W3C  SWS-IG 


Thompson,  L.  F.,  &  Gillan,  D.  J.  (2010).  Chapter  5.  Social  Factors  in  Fluman-Robot  Interaction.  In  M.  Barnes  and 
F.  Jentsch  (Eds.),  Human-Robot  Interactions  in  Future  Military  Operations.  London:  Ashgate  Publishing,  Inc. 

Key  Words: 
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Book  Chapter/Section 

Summary  abstract: 

Much  of  this  book  centers  on  technical,  usability  issues  pertaining  to  humans’  interactions  with  robots.  Although 
usability  is  a  critical  antecedent  of  technology  acceptance,  it  is  by  no  means  the  only  determinant.  Even  an  intuitive, 
easy-to-use  robot  will  be  disregarded  if  military  personnel  dislike  it,  distrust  it,  doubt  it,  or  resent  its  ability  to 
contribute  effectively  and  efficiently  to  the  mission  at  hand.  Burke,  Coovert,  Murphy,  Riley,  and  Rogers  (2006) 
outlined  four  major  HRI  issues  requiring  attention  from  human  factors  professionals.  Included  on  this  list  was  the 
need  to  consider  the  factors  driving  acceptance  of  robots  in  work  settings.  The  failure  to  accept  a  robotic  assistant 
can  result  in  situations  where  operators  or  soldiers  sharing  a  field  of  operations  with  robots  (1)  do  not  place  enough 
weight  on  the  information  or  suggestions  provided  by  the  robot,  (2)  refuse  to  interact  with  the  robot  altogether,  or  (3) 
actively  reject  or  react  counter  to  the  information  from  the  robot  (Madhavan,  Wiegmann,  &  Lacson,  2006). 

The  purpose  of  this  chapter  is  to  discuss  the  social  factors  in  HRI,  especially  as  they  affect  the  acceptance  of  robots 
and  their  integration  into  military  teams.  Implicit  in  the  approach  to  this  chapter  is  a  broad  use  of  the  term 
“interface.”  Our  use  of  “interface”  covers  any  nexus  of  interaction  between  a  human  and  robot,  including  both  the 
means  by  which  operators  control  and  monitor  the  robot  and,  most  importantly,  the  points  of  contacts  between 
soldiers  in  the  field  and  robots.  Thus,  the  interface  may  involve  social,  as  well  as  cognitive  and  perceptual 
interactions.  With  this  in  mind,  we  begin  this  chapter  by  considering  the  dynamics  that  may  influence  the  acceptance 
of  robots  by  individual  and/or  teams  of  soldiers.  We  then  discuss  a  number  of  issues  that  emerge  as  robots  are 
designed  to  interact  with  soldiers  in  social  terms  normally  associated  with  human-to-human  exchanges. 


Tilak,  R.,  Xholi,  I.,  Schowalter,  D.,  Ferris,  T.,  Hameed,  S.,  &  Sarter,  N.  (2008).  Crossmodal  Links  in  Attention  in 
the  Driving  Environment:  The  Roles  of  Cueing  Modality,  Signal  Timing,  and  Workload.  In  In  Proceedings  of  52nd 
Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  22-26  September  2008.  New  York  City,  NY  USA. 
New  York  City,  NY  USA:  HFES. 
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Summary  abstract: 

Multimodal  information  presentation  has  been  proposed  as  a  means  to  support  timesharing  in  complex  data-rich 
environments.  To  ensure  the  effectiveness  of  this  approach,  it  is  necessary  to  consider  performance  effects  of 
recently  discovered  crossmodal  spatial  and  temporal  links  in  attention,  as  well  as  their  interaction  with  other 
performance-shaping  factors.  The  main  goals  of  this  research  were  to  confirm  that  performance  effects  of 
crossmodal  links  in  spatial  attention  scale  to  complex  environments  and  to  examine  how  these  effects  vary  as  a 
function  of  cue  modality,  signal  timing,  and  workload.  In  the  present  study,  set  in  a  driving  simulation,  spatially 
valid  and  invalid  auditory  and  tactile  cues  preceded  the  presentation  of  visual  targets  at  various  stimulus-onset 
asynchronies  and  under  different  levels  of  workload  induced  by  simulated  wind  gusts  of  varied  intensity.  The 
findings  from  this  experiment  confirm  that  visual  target  identification  accuracies  and  response  times  are,  overall, 
more  accurate  and  faster  when  validlycued.  Significant  interactions  were  found  between  cue  validity,  stimulus  onset 
asynchrony  (SOA),  and  cue  modality,  such  that  valid  tactile  cueing  is  most  beneficial  at  shorter  (100  -  200  ms) 
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SOAs,  while  valid  auditory  cueing  resulted  in  faster  responses  than  invalid  cueing  at  500  ms  SOAs,  but  slower 
responses  at  1000  ms  SOAs.  Tactile  error  rates  were  significantly  higher  than  auditory  error  rates  at  various 
interactions  of  modality  and  SOA.  These  findings  were  robust  across  all  workload  conditions.  They  highlight  the 
need  for  context-sensitive  information  presentation  and  can  inform  the  design  of  multimodal  interfaces  for  a  wide 
range  of  application  domains. 


Tinapple,  D.,  &  Woods,  D.  (2003).  Message  Overload  from  the  Inbox  to  Intelligence  Analysis:  How  Spam  and 
Blogs  Point  to  New  Tools.  In  In  Proceedings  of  47th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society 
Conference.  Denver,  CO. 
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Summary  abstract: 

Patterns  of  responses  to  “message  overload"  can  be  seen  in  the  ways  in  which  people  adapt  messaging  systems  and 
capabilities.  Blogging  is  an  effective  and  increasingly  popular  decentralized  form  of  group  communication  that  is 
proving  useful  in  helping  people  find  and  share  what  is  informative.  We  look  to  blogging  for  clues  to  new  solutions 
to  the  problem  of  “data  overload”  in  the  world  of  email.  These  design  solutions  to  email  overload  go  beyond  efforts 
to  block  spam,  and  are  based  on  shifting  the  basic  unit  of  organization  toward  communication  relationships  that 
allow  patterns  in  communications  to  emerge. 


Tittle,  J.  S.,  Roesler,  A.,  &  Woods,  D.  D.  (2002).  The  Remote  Perception  Problem.  In  In  Proceedings  of  46th  Annual 
Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Baltimore,  MD. 
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Summary  abstract: 

Previous  research  (e.g.,  Casper,  2002;  Darken,  Kempster,  &  Peterson,  2001)  has  shown  that  observers  demonstrate 
poor  spatial  awareness  based  on  video  provided  from  remote  environments.  Such  a  result  is  understandable  given 
that  remote  vision  systems  provide  impoverished  representations  that  leave  out  higher  order  cues  essential  to  build 
coherent  percepts  and  models  of  the  world  being  explored.  If  tele-presence  or  remote  vision  is  to  be  useful  in  the 
future,  the  raw  video  needs  to  somehow  be  augmented  to  recover  what  was  lost  by  decoupling  the  human  perceptual 
processor  from  the  natural  environment. 


Tittle,  J.  S.,  Woods,  D.  D.,  Roesler,  A.,  Howard,  M.,  &  Phillips,  F.  (2001).  The  Role  of2-D  and  3-D  Task 
Performance  in  the  Design  and  Use  of  Visual  Displays.  In  In  Proceedings  of  45th  Annual  Meeting  of  the  Human 
Factors  and  Ergonomics  Society.  Minneapolis,  MN. 
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Summary  abstract: 

Recent  advances  in  computerized  graphics  capabilities  have  led  to  increased  development  and  use  of  3-D  displays. 

In  this  paper  we  will  discuss  a  survey  of  results  from  the  visual  perception  literature  that  compares  3-D  and  2-D  task 
performance.  The  basic  trend  of  this  research  suggests  that  designers  of  visual  displays  must  recognize  that  the 
internal  representations  observers  extract  from  3-D  displays  will  be  distinctly  different,  and  perhaps  more 
qualitative,  than  the  computer  representations  or  models  used  to  generate  these  displays.  Finally,  the  surveyed 
literature  indicates  that  crucial  information  in  3-D  displays  should  not  be  encoded  as  differences  in  magnitudes  of 
depth,  orientation,  or  curvature.  Instead  designers  should  try  to  encode  important  task  dimensions  within  more 
qualitative  3-D  distinctions  such  as  ordinal  depth,  deviations  from  co-planarity,  and  the  presence  of  flat  versus 
curved  surfaces. 
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Toninelli,  A.,  Bradshaw,  J.  M.,  Kagal,  L.,  &  Montanari,  R.  (2006).  Rule-Based  and  Ontology-Based  Policies: 
Toward  a  Hybrid  Approach  to  Control  Agents  in  Pervasive  Environments.  In  In  Proceedings  of  International 
Semantic  Web  Conference  Policy  Workshop  (ISWC  05),  Dublin,  Ireland. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Policies  are  being  increasingly  used  for  controlling  the  behavior  of  complex  multi-agent  systems.  The  use  of  policies 
allows  administrators  to  regulate  agent  behavior  without  changing  source  code  or  requiring  the  consent  or 
cooperation  of  the  agents  being  governed.  However,  policy-based  control  can  sometimes  encounter  difficulties  when 
applied  to  agents  that  act  in  pervasive  environments  characterized  by  frequent  and  unpredictable  changes.  In  such 
cases,  we  cannot  always  specify  policies  a  priori  to  handle  any  operative  run  time  situation,  but  instead  require 
continuous  adjustments  to  allow  agents  to  behave  in  a  contextually  appropriate  manner.  To  address  these  issues, 
some  policy  approaches  for  governing  agents  in  pervasive  environments  specify  policies  in  a  way  that  is  both 
context-based  and  semantically-rich.  Two  approaches  have  been  used  in  recent  research:  an  ontology-based 
approach  that  relies  heavily  on  the  expressive  features  of  Description  Logic  (DL)  languages,  and  a  rule-based 
approach  that  encodes  policies  as  Logic  Programming  (LP)  rules.  The  aim  of  this  paper  is  to  analyze  the  emerging 
directions  for  the  specification  of  semantically-rich  context-based  policies,  highlighting  their  advantages  and 
drawbacks.  Based  on  our  analysis  we  describe  a  hybrid  approach  that  exploits  the  expressive  capabilities  of  both  DL 
and  LP  approaches. 


Tonti,  G.,  Montanari,  R.,  Bradshaw,  J.  M.,  Bunch,  L.,  Jeffers,  R.,  Suri,  N.,  &  Uszok,  A.  (2004).  Automated 
Generation  of  Enforcement  Mechanisms  for  Semantically- Rich  Policies  in  Java-Based  Multi- Agent  Systems.  In  In 
Proceedings  of  First  IEEE  Symposium  on  Multi-Agent  Security  and  Survivability  (MAS&S  2004). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Policies  are  being  increasingly  used  for  controlling  the  behavior  of  complex  systems  (including  agent  systems).  The 
use  of  policies  allows  administrators  to  specify  agent  permissions  and  obligations  without  changing  source  code  or 
requiring  the  consent  or  cooperation  of  the  entities  being  governed.  Past  approaches  to  policy  representation  have 
been  restrictive  in  many  ways.  By  way  of  contrast,  semantically-rich  policy  representations  can  reduce  human  error, 
simplify  policy  analysis,  reduce  policy  conflicts,  and  facilitate  interoperability.  However,  semantically-rich  policies 
increase  the  complexity  of  fielding  policy-governed  multi-agent  systems.  This  paper  discusses  some  technical 
challenges  to  automatically  enforce  semantically-rich  security  policies  in  Java-based  multi-agent  systems  and 
presents  an  engineering  approach  for  addressing  some  of  these  challenges.  We  have  developed  a  first 
implementation  that  allows  to  enforce  OWL  policies  represented  using  the  KAoS  policy  framework  into  multi-agent 
systems  built  on  top  of  the  JDK1.4.  The  proposed  solution  allows  to  control  the  behavior  of  agents  at  a  high  level  of 
abstraction  and  exploits  the  security  mechanisms  provided  by  the  Java  Authentication  and  Authorization  Service 
(JAAS)  to  enforce  OWL  policies. 


Tonti,  G.,  Bradshaw,  J.  M.,  Jeffers,  R.,  Montanari,  R.,  Suri,  N.,  &  Uszok,  A.  (2004).  Semantic  Web  Languages  for 
Policy  Representation  and  Reasoning:  A  Comparison  of  KAoS,  Rei,  and  Ponder.  In  In  Proceedings  of  International 
Semantic  Web  Conference  (ISWC  03).  Sanibel  Island,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Policies  are  being  increasingly  used  for  automated  system  management  and  controlling  the  behavior  of  complex 
systems,  allowing  administrators  to  modify  system  behavior  without  changing  source  code  or  requiring  the  consent 
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or  cooperation  of  the  components  being  governed.  Past  approaches  to  policy  representation  have  been  restrictive  in 
many  ways.  By  way  of  contrast,  semantically-  rich  policy  representations  can  reduce  human  error,  simplify  policy 
analysis,  reduce  policy  conflicts,  and  facilitate  interoperability.  In  this  paper,  we  compare  three  approaches  to  policy 
representation,  reasoning,  and  enforcement.  We  highlight  similarities  and  differences  between  Ponder,  KAoS,  and 
Rei,  and  sketch  out  some  general  criteria  and  properties  for  more  adequate  approaches  to  policy  semantics  in  the 
future. 


Trent,  S.  A.,  Patterson,  E.  S.,  &  Woods,  D.  D.  (2007).  Challenges  for  Cognition  in  Intelligence  Analysis.  Journal  of 
Cognitive  Engineering  and  Decision  Making,  7(1),  75-97. 
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Summary  abstract: 

Intelligence  analysis  is  a  high-stakes  domain  that  poses  challenges  to  effective  individual  and  collaborative 
cognition.  The  design  of  support  tools  and  analytical  pedagogy  could  benefit  from  an  understanding  of  how 
challenges  that  are  reported  in  other  decision-making  literature  generalize  and  are  manifested  in  more  naturalistic 
settings.  The  objective  of  this  research  was  to  elicit  challenges  for  cognition  in  collaborative  intelligence  analysis. 
Two  complementary  research  methods  were  used:  unstructured  interviews  with  46  analysts  and  supervisors,  and 
observations  of  eight  teams  of  military  intelligence  analysts  conducting  a  training  scenario.  Interviews  with 
designers,  educators,  and  practitioners  in  the  intelligence  community  revealed  trends  in  unsupported  cognitive  work 
and  cultural  challenges,  whereas  observations  from  a  training  exercise  for  army  intelligence  analysts  instantiated 
other  cognitive  challenges  of  collaborative  analysis.  This  study  indicates  that  analytical  style  (part  tradition  and  part 
due  to  individual  reasoning  tendencies)  can  result  in  premature  narrowing,  difficulty  in  reframing,  and  getting  lost  in 
the  details.  The  study  also  illustrates  the  effects  of  friction  within  and  across  federated  teams,  how  variable  tempo 
can  produce  inexpert  behavior,  and  considerations  for  the  design  of  analytical  support  tools.  This  work  suggests  the 
value  of  complementary  research  methods  in  the  study  of  other  domains  involving  collaborative  work.  It  is  likely 
that  these  cognitive  challenges  affect  other  domains  involving  collaborative  analysis.  Finally,  this  study  suggests 
that  the  effects  of  individual  cognitive  challenges  are  difficult  to  isolate  in  naturalistic  settings  and  should  most 
likely  be  considered  collectively  rather  than  independently. 

Know  the  enemy  and  know  yourself;  in  a  hundred  battles  you  will  never  be  in  peril.  -  Sun  Tzu 


Trent,  S.  A.,  Smith,  M.  W.,  Zelik,  D.  J.,  Grossman,  J.  B.,  &  Woods,  D.  D.  (2009).  Chapter  17.  Reading  Intent  and 
Other  Cognitive  Challenges  in  Intelligence  Analysis.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision 
Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  III.  Acting  on  Battlefield  Information: 
48hrBooks  by  Alion  Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the 
Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research 
Management  Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1100 
New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr- 
webtest. alionscience.com/AdaCta/ Agenda,  aspx. 
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Summary  abstract: 

Utilizing  information  for  planning  and  decision  making  entails  a  process  of  analysis,  which  is  subject  to  particular 
vulnerabilities  given  the  demands  confronting  the  analyst.  In  the  context  of  military  operations,  intelligence  analysis 
is  a  complex  process  involving  many  cognitive  challenges.  This  project,  through  a  series  of  studies,  examined  the 
cognitive  challenges  confronting  intelligence  analysts  in  their  work. 

Over  the  course  of  the  project,  three  important  themes  have  emerged: 
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First,  capturing  cognitive  challenges  of  intelligence  analysis  requires  methodological  innovations.  Requirements 
development  methods  which  only  look  at  normative  "textbook"  processes  will  fail  to  identify  the  challenges  arising 
from  complex  situations,  time  pressures,  resource  limitations,  and  coping  strategies.  More  naturalistic,  ecologically 
valid  methods  are  needed.  However,  the  security  constraints  present  in  the  intelligence  domain  require  further 
innovations,  such  as  more  efficient  utilization  of  limited  access  to  the  domain,  and  the  use  of  non-classified  analog 
scenarios.  Multiple  studies  using  complex  but  open  source  scenarios  with  professional  analysts  balanced  these 
constraints  in  order  to  gain  insight  into  cognitive  challenges  inherent  in  intelligence  analysis. 

Second,  assessing  analytic  performance  is  best  accomplished  with  a  structured  yet  adaptive  model  of  the  process. 
What  constitutes  expert  analytic  performance  varies  depending  on  demands,  resources,  the  nature  of  the  problem, 
and  the  needs  of  the  decision  makers.  No  simple  algorithmic  approach  can  adequately  reflect  the  necessary  trade¬ 
offs  expert  analysts  make.  Therefore,  we  studied  how  professional  analysts  decide  what  is  sufficiently  rigorous  in 
context,  and  developed  a  metric  for  assessing  the  rigor  of  analytic  processes. 

Third,  reading  the  intent  of  others  is  both  a  core  function  of  analysis,  and  a  fundamental  cognitive  challenge  in 
understanding  complex  social  systems.  Forecasting  how  a  group  will  act  in  the  future  (including  how  the  group 
might  react  to  a  Course  Of  Action)  involves  understanding  the  group's  intentions.  However,  the  research  showed 
reading  the  intent  of  other  groups  accurately  is  complex  and  error  prone. 

The  series  of  studies  look  at  the  challenges  of  intelligence  analysis  with  respect  to  the  following: 

•  The  practice  of  analysis  in  real  situations,  with  multiple  teams,  production  demands,  and  environmental  pressures 
(in  other  words,  the  Where,  When,  and  with  Whom  of  analysis) 

•  Reflections  on  the  process  of  analysis,  and  interaction  between  analysis  and  supervisors/decision-makers  (the  How 
and  for  Whom) 

•  Analysis  of  other  groups'  structures  and  intentions  (the  What  and  Why) 


Trent,  S.  A.,  Voshell,  M.,  Grossman,  J.,  Schoenwald,  J.,  Patterson,  E.,  Goldstein,  S.,  Scherbarth,  M.,  Tittle,  J.,  & 
Dominguez,  C.  (2007).  Federated  Observational  Research:  A  Unique  Investigation  Strategy  for  the  Study  of 
Distributed  Cognition.  In  In  Proceedings  of  K.Mosier  &  U.  Fischer  (eds.),  Proceedings  of  the  Eighth  International 
Conference  on  Naturalistic  Decision  Making. 
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ethnography,  team  cognition,  observational  research,  intelligence  analysis. 

Summary  abstract: 

The  study  of  team  cognition  in  complex  domains  is  typically  hampered  by  two  aspects  -  the  concurrent,  distributed 
nature  of  the  cognitive  work  and  the  complexities  of  domain-specific  tasks.  Researchers  unfamiliar  with  the  work 
prior  to  observation  may  overlook  critical  vulnerabilities  and  interactions  from  simultaneous  processes  in  multiple 
areas.  In  this  study,  an  interdisciplinary  observation  team  used  unique  variations  on  established  ethnographic 
techniques.  Rather  than  design  a  scaled  world  study,  the  research  team  first  participated  in  a  counter-insurgency 
intelligence  training  operation.  This  afforded  a  firsthand  appreciation  for  critical  tasks  and  operational  language  of 
the  practitioners.  The  team  then  used  a  federated  observation  technique  to  simultaneously  observe  four  independent 
squads  of  analysts  conduct  the  same  exercise.  This  paper  outlines  our  methods  and  discusses  some  initial  findings 
that  suggest  implications  for  support  tool  design,  collaborative  work  and  analytical  pedagogy.  It  concludes  with  the 
benefits  and  limitations  of  our  investigation  strategy. 


Troberg,  E.,  &  Gillan,  D.  J.  (2007).  Measuring  Spatial  Knowledge:  Effects  of  the  Relation  between  Acquisition  and 
Testing.  In  In  Proceedings  of  51st  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Baltimore,  MD. 
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Summary  abstract: 

Performance  in  human-robot  interaction  is  related  to  the  operators  mental  map  of  the  space  in  which  the  robot 
travels.  Accordingly,  accurate  assessment  of  mental  maps  will  be  important  for  the  design  of  human-robot 
interfaces.  The  present  research  used  a  factorial  design  experiment  to  examine  two  methods  for  acquiring  spatial 
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knowledge  (reading  a  map  vs.  navigating  in  the  space),  three  methods  of  testing  spatial  knowledge  (drawing  a  map, 
navigating  through  the  space,  and  estimating  point-to-point  distances.  The  results  showed  that  performance  in  the 
navigation  test  was  influenced  by  factors  unrelated  to  the  navigated  distance,  whereas  map  drawing  especially  was 
closely  related  to  the  actual  distance.  Map  drawing  resulted  in  a  stronger  relation  between  map  distance  and  actual 
distance  in  the  map  training  condition  than  in  the  navigation  training  condition.  The  results  are  interpreted  in  terms 
of  transfer  appropriate  processing,  and  are  applied  to  human-robot  interface  design. 


Turner,  G.  A.,  Wang,  G.,  &  Winkler,  R.  (2003).  An  Agent-Based  Shared  Display  Architecture.  In  In  Proceedings  of 
Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the  Objective 
Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S.  Army  Research 
Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

Real-time  information  sharing  is  a  critical  factor  in  today’s  and  tomorrow’s  battlefield  and  the  Future  Combat 
Systems.  In  this  article,  we  propose  a  shared  display  architecture  based  on  a  software  agent  technique,  which  enables 
real-time  sharing  of  large  quantity  GIS  and  tactical  data  among  heterogeneous  display  devices  in  a  network 
environment.  We  define  the  architecture  by  specifying  the  roles  of  the  agent  and  the  contributions  it  provides  to  the 
solution.  We  also  offer  an  opinion  comparing  this  agent-based  approach  with  the  traditional  database  mechanisms 


Tuzar,  D.,  &  Woods,  D.  D.  (2006).  Reorientation  in  Dynamic  Situations,  tuzar.l@osu.edu  woods.2@osu.edu 
http://csel.eng.ohio-state.edu. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

The  activities  of  monitoring  and  controlling  in  event  driven  environments  concerns  in  parallel  tactical  tasks  as  well 
as  strategic  tasks.  Dynamic  control  environments  often  are  mixed  environments,  real  world  and/or  virtual  worlds. 
Strategic  and  tactical  operations  are  either  perceived  “through  a  window”  showing  a  sector  of  a  real  world 
environment  (real  world),  or  virtually  showing  a  virtual  representation  on  a  computer  display  (virtual  world). 
Problem  holders  must  focus  their  perceptual  system,  shift  priorities,  control  tasks  and  sub  tasks,  and  time  share 
between  tasks  to  succeed  within  such  high  density  environments.  Agents  in  artificial  and  high  pace  environments 
experience  workload  bottlenecks  and  data  overload  leading  to  critical  situations  or  to  failures  of  a  complex  system. 
This  topic  landscape  explores  the  operators’  difficulties  in  interacting  with  Human-Machine-Interfaces  (HMI)  for 
tactical  tasks  within  event  driven  worlds,  develops  a  conceptual  space  for  reorientation,  proposes  a  frame  of 
reference  for  evaluating  re -orienting  costs,  suggests  design  concepts,  and  relates  findings  to  attention. 


Uszok,  A.,  &  Bradshaw,  J.  M.  (2008).  Demonstrating  Selected  W3c  Policy  Languages  Interest  Group  Use  Cases 
Using  the  KAoS  Policy  Services  Framework.  In  In  Proceedings  of  2008  IEEE  Conference  on  Policy,  Palisades,  NY. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
KAoS,  policy  use  cases,  PLING,  W3C 

Summary  abstract: 

This  demonstration  will  present  policies  and  scenarios  from  selected  W3C  Policy  Language  Group  use  cases.  The 
flexibility  of  the  KAoS  ontology-based  policy  services  framework  will  be  demonstrated  by  modeling  the  very 
diverse  policies  described  in  the  use  cases.  The  integration  of  the  KAoS  policy  decision  point  (Guard)  with  the 
JBoss  server  will  be  shown.  One  of  the  use  cases  requires  spatial  reasoning  for  location-based  policies. 
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Uszok,  A.,  Bradshaw,  J.  M.,  Hayes,  P.,  Jeffers,  R.,  Johnson,  M.,  Kulkarni,  S.,  Breedy,  M.,  Lott,  J.,  &  Bunch,  L. 
(2004).  DAML  Reality  Check:  A  Case  Study  of  KAoS  Domain  and  Policy  Services.  In  In  Proceedings  of  the 
International  Semantic  Web  Conference  (ISWC  03).  Sanibel  Island,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Description-logic-based  knowledge  representations  and  reasoning  methods  are  being  used  increasingly  as  the  basis 
for  semantically-rich  software  services.  Using  such  representations  and  reasoning  methods  in  comprehensive 
applications  is  among  one  of  the  best  ways  to  identify  and  understand  gaps  and  limitations.  KAoS  domain  and 
policy  services,  which  rely  extensively  on  a  DAML-based  ontology,  are  used  as  a  case  study  to  investigate  an  in- 
depth  application  of  DAML.  In  this  context,  we  explore  the  current  limitations  of  DAML  semantics.  We  also 
describe  our  observations  about  the  place  of  ontology  description  at  the  heart  of  KAoS  services,  and  outline 
requirements  for  effective  interfaces  for  humans  and  for  translation  to  and  from  efficient  programming 
environments.  Next,  we  present  the  problems  implicated  by  development  of  ontology  descriptions  in  a  distributed 
dynamic  environment.  Finally,  we  assess  the  utility  and  maturity  of  available  tools  such  as  editors,  programming 
libraries  and  inference  engines.  An  application  of  KAoS  to  provide  policies  for  DAML-S  Web  services  concludes 
the  paper. 


Uszok,  A.,  Bradshaw,  J.  M.,  Jeffers,  R.,  Johnson,  M.,  Tate,  A.,  Dalton,  J.,  &  Aitken,  S.  (2004).  Policy  and  Contract 
Management  for  Semantic  Web  Services.  In  In  Proceedings  of  AAAI  Spring  Symposium.  Stanford,  CA.,  and 
Proceedings  of  IEEE  Intelligent  Systems,  (July/ August),  pp.  26-35. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

This  paper  summarizes  our  efforts  to  develop  capabilities  for  policy  and  contract  management  for  Semantic  Web 
Services  applications.  KAoS  services  and  tools  allow  for  the  specification,  management,  analyzes,  disclosure  and 
enforcement  of  policies  represented  in  OWL.  We  discuss  three  current  Semantic  Web  Services  applications  as 
examples  of  the  kinds  of  roles  that  a  policy  management  framework  can  play:  as  an  authorization  service  in  grid 
computing  environments,  as  a  distributed  policy  specification  and  enforcement  capability  for  a  semantic 
matchmaker,  and  as  a  verification  tool  for  services  composition  and  contract  management. 


Uszok,  A.,  Bradshaw,  J.  M.,  Jeffers,  R.,  Tate,  A.,  &  Dalton,  J.  (2005).  Applying  KAoS  Services  to  Ensure  Policy 
Compliance  for  Semantic  Web  Services  Workflow  Composition  and  Enactment.  In  In  Proceedings  of  International 
Semantic  Web  Conference  (ISWC  2004).  Hiroshima,  Japan. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  this  paper  we  describe  our  experience  in  applying  KAoS  services  to  ensure  policy  compliance  for  Semantic  Web 
Services  workflow  composition  and  enactment.  We  are  developing  these  capabilities  within  the  context  of  two 
applications:  Coalition  Search  and  Rescue  (CoSAR-TS)  and  Semantic  Firewall  (SFW).  We  describe  how  this  work 
has  uncovered  requirements  for  increasing  the  expressivity  of  policy  beyond  what  can  be  done  with  description  logic 
(e.g.,  role-value-maps),  and  how  we  are  extending  our  representation  and  reasoning  mechanisms  in  a  carefully 
controlled  manner  to  that  end.  Since  KAoS  employs  OWL  for  policy  representation,  it  fits  naturally  with  the  use  of 
OWL-S  workflow  descriptions  generated  by  the  AIAI I-X  planning  system  in  the  CoSARTS  application.  The 
advanced  reasoning  mechanisms  of  KAoS  are  based  on  the  JTP  inference  engine  and  enable  the  analysis  of  classes 
and  instances  of  processes  from  a  policy  perspective.  As  the  result  of  analysis,  KAoS  concludes  whether  a  particular 
workflow  step  is  allowed  by  policy  and  whether  the  performance  of  this  step  would  incur  additional  policy- 
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generated  obligations.  Issues  in  the  representation  of  processes  within  OWL-S  are  described.  Besides  what  is  done 
during  workflow  composition,  aspects  of  policy  compliance  can  be  checked  at  runtime  when  a  workflow  is  enacted. 
We  illustrate  these  capabilities  through  two  application  examples.  Finally,  we  outline  plans  for  future  work. 


Uszok,  A.,  Bradshaw,  J.  M.,  Lott,  J.,  Breedy,  M.,  Bunch,  L.,  Feltovich,  P.,  Johnson,  M.,  &  Jung,  H.  (2008).  New 
Developments  in  Ontology-Based  Policy  Management:  Increasing  the  Practicality  and  Comprehensiveness  of  KAoS 
In  In  Proceedings  of  2008  IEEE  Conference  on  Policy,  Palisades,  NY. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

policy,  ontology,  OWL,  KAoS,  policy 

management 

Summary  abstract: 

The  KAoS  policy  management  framework  pioneered  the  use  of  semantically-rich  ontological  representation  and 
reasoning  to  specify,  analyze,  deconflict,  and  enforce  policies  [9,  10].  The  framework  has  continued  to  evolve  over 
the  last  five  years,  inspired  by  both  technological  advances  and  the  practical  needs  of  its  varied  applications.  In  this 
paper,  we  describe  how  these  applications  have  motivated  the  partitioning  of  components  into  a  well-defined  three- 
layer  policy  management  architecture  that  hides  ontology  complexity  from  the  human  user  and  from  the  policy- 
governed  system.  The  power  of  semantic  reasoning  is  embedded  in  the  middle  layer  of  the  architecture  where  it  can 
provide  the  most  benefit.  We  also  describe  how  the  policy  semantics  of  the  core  KAoS  Policy  Ontology  has  grown 
in  its  comprehensiveness.  The  flexible  and  mature  architecture  of  KAoS  enables  straightforward  integration  with  a 
variety  of  deployment  platforms,  ranging  from  highly  distributed  systems,  such  as  the  AFRL  Information 
Management  System,  to  human-robotic  interaction,  to  dynamic  management  of  quality-of-service  and  crossdomain 
information  management  of  wireless  networks  in  resource-constrained  or  security-sensitive  environments. 


Vanni,  M.,  Voss,  C.  R.,  &  Tate,  C.  (2004).  Ground  Truth,  Reference  Truth  &  “Omniscient  Truth”  —  Parallel  Phrases 
in  Parallel  Texts  for  Mt  Evaluation.  In  In  Proceedings  of  Workshop  on  The  Utility  of  Parallel  and  Comparable 
Corpora,  Conference  on  Language  Resources  and  Evaluation  (LREC),  Lisbon,  Portugal.  [Refereed]. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Recently  introduced  automated  methods  of  evaluating  machine  translation  (MT)  systems  require  the  construction  of 
parallel  corpora  of  source  language  (SL)  texts  with  human  reference  translations  in  the  target  language  (TL).  We 
present  a  novel  method  of  exploiting  and  augmenting  these  resources  for  task-based  MT  evaluation,  assessing  how 
accurately  people  can  extract  Who,  When,  and  Where  elements  of  information  from  TL  output  texts  of  different  MT 
engines.  This  paper  reports  on  the  first  phase  of  our  research  establishing  a  baseline  MT  evaluation  process  with  (i) 
the  construction  and  (ii)  the  annotation  and  inter-annotator  rates  of  an  annotated  extraction  corpus,  and  (iii)  our 
results  applying  the  corpus  in  the  evaluation  of  three  Arabic-to-English  MT  engines.  In  this  corpus,  the  elements  of 
interest  are  identified  as  parallel  phrases  across  the  parallel  texts  of  the  SL,  the  reference  translations,  and  the  MT 
engine  outputs,  where  they  are  annotated  and  called,  respectively  the  Ground  Truth  (GT),  Reference  Truth  (RT),  and 
Omniscient  Truth  (OT)  items  in  the  parallel  texts.  Our  evaluation  of  three  MT  engines  with  the  corpus  yields 
precision  and  recall  accuracy  measures  that,  together  with  a  loss  measure,  clearly  rank  the  engines  and,  unlike  other 
evaluation  metrics,  indicate  diagnostically  where  output  improvements  will  assist  on  extraction. 


Veinott,  B.,  Cox,  D.,  &  Mueller,  S.  (2009).  Social  Media  Supporting  Disaster  Response:  Evaluation  of  a 
Lightweight  Collaborative  Tool.  In  In  Proceedings  of  Naturalistic  Decision  Making  9.  London,  UK. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Collaboration,  Social  Media,  Design,  Virtual  Ethnography,  Disaster  Response 
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Summary  abstract: 

Motivation:  Disaster  response  efforts  rely  heavily  on  ad  hoc,  cross  organization  teams  with  multiple  stakeholders, 
who  must  collaboratively  make  sense  of  the  situation,  and  are  often  working  at  a  distance.  These  properties  make  it 
difficult  to  design  collaborative  tools  in  support  of  decision  making  in  these  environments. 

Research  Approach:  In  this  paper  we  use  virtual  ethnographic  techniques  as  a  way  to  bridge  the  gap  between 
important  contextual  and  socio-technical  conditions  and  early  tool  assessment.  For  an  example  case  of  some  socio- 
technical  issues,  we  discuss  Twitter  use  during  the  San  Diego,  California  wildfires  in  October  2007. 

Findings:  We  found  that  Twitter  supported  some  aspects  of  distributed  collaboration  well,  but  not  others. 
Originality:  This  research  provided  an  early  examination  of  the  collaboration  this  form  of  connectivity  supports 
which  can  be  applied  in  several  domains. 


Veinott,  E.  S.,  Klein,  G.  A.,  &  Wiggins,  S.  L.  (2010).  Evaluating  the  Effectiveness  of  the  Premortem  Technique  on 
Plan  Confidence. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Report 
Summary  abstract: 

One  problem  affecting  crisis  management  planning  teams  is  overconfidence  -  an  inflated  belief  that  a  plan  will  be 
successful.  In  this  paper  we  compared  the  effect  of  several  different  methods  for  reducing  individual  team  member 
confidence  levels  (Critique,  Pros-Cons  generation,  Cons  only  generation,  and  a  PreMortem)  and  compared  each  to  a 
baseline  control  condition.  One  hundred  and  seventy-eight  people  participated  in  one  of  five  conditions  to  evaluate 
an  H1N1  plan  in  a  university  context.  Participants  provided  several  ratings  of  confidence  in  the  plan’s  success  over 
the  course  of  the  plan  evaluation,  generated  reasons  during  their  plan  evaluation  and  provided  subsequent 
confidence  and  understanding  ratings.  We  compared  several  techniques  commonly  used  (e.g.,  Critique,  Devil’s 
Advocate,  Pro-Cons)  and  a  newer  technique,  the  PreMortem.  The  PreMortem  technique  (imagining  that  a  plan  has 
failed  and  then  trying  to  explain  why)  reliably  reduced  confidence  levels  more  than  the  other  conditions,  and  may  be 
useful  in  combating  overconfidence  in  crisis  management  planning. 


Voshell,  M.,  &  Oomes,  A.  H.  J.  (2006).  Coordinating  (Shared)  Perspectives  in  Robot  Assisted  Search  &  Rescue.  In 
In  Proceedings  of  the  3rd  International  ISCRAM  Conference  (B.  Van  de  Walle  and  M.  Turoff,  eds.),  Newark,  NJ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Urban  search  and  rescue,  human-robot  interaction,  remote  perception 

Summary  abstract: 

From  high  fidelity  field  exercises  to  disaster  response  deployments,  search  and  rescue  robots  are  being  readily 
integrated  into  rescue  operations.  Previous  research  has  proposed  that  for  such  new  technology  to  be  successful  in  an 
operation  the  organization  architecture  needs  to  support  the  coordination  of  shared  perspectives  between  the  human 
team  members  and  the  robotic  platforms.  For  this,  the  robot  platforms  need  to  be  effective  team  players  in  the  field 
of  practice.  Based  on  this  conceptual  model,  this  paper  introduces  a  novel  software  interface  utilizing  virtual 
position  and  orientation  indicators  to  alleviate  perceptual  ambiguities  and  navigation  problems  experienced  by  robot 
handlers  and  problem  holders.  By  actively  orchestrating  and  sharing  these  indicators  between  handler  and  operator 
displays,  the  interface  caters  to  user  expertise  and  to  the  natural  competency  of  the  human  perceptual  system.  These 
probes  provide  a  basic  tool  for  aiding  robot  navigation  and  way-finding  fundamental  to  effective  team  coordination 
and  communication  in  urban  search  and  rescue  missions. 


Voshell,  M.,  Trent,  S.,  Prue,  B.,  &  Fern,  L.  C.  (2008).  Cultivating  Resilience  in  Urban  Fire  Fighting:  Supporting 
Skill  Acquisition  through  Scenario  Design.  In  In  Proceedings  of  52nd  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society.  22-26  September  2008.  New  York  City,  NY  USA.  New  York  City,  NY  USA:  HFES. 
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Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Whether  studying  human  cognition,  designing  new  technologies,  or  exploring  concepts  such  as  safety  and  resilience, 
understanding  decision  making  in  natural  settings  is  fundamental  to  Cognitive  Systems  Engineering  (CSE).  Across 
all  of  these,  it  is  crucial  for  human  factors  engineers  to  be  able  to  create  realistic  scenarios  of  interest  that  hit  upon 
“cognitive  pressure  points”  (Woods  and  Dekker,  2000)  to  elicit  challenges  and  observe  patterns  in  cognitive  work. 
Although  many  converging  methodologies  have  provided  a  rich  ecological  body  of  practice-based  research  and 
application,  there  is  a  shortage  of  research  demonstrating  the  efficacy  of  scenario-based  design  in  support  of  skill 
acquisition.  Based  on  continuing  research  with  a  major  metropolitan  fire  department,  the  current  paper  reports  initial 
findings  from  ongoing  work  analysis.  From  a  series  of  scenarios,  patterns  of  coordination  challenges  are  discussed 
with  implications  for  future  emergency  response  operations  and  training. 


Voshell,  M.,  Woods,  D.  D.,  &  Phillips,  F.  (2005).  Overcoming  the  Keyhole  in  Fluman-Robot  Interaction:  Simulation 
and  Evaluation.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando, 
FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

When  environment  access  is  mediated  through  robotic  sensors,  field  experience  and  naturalistic  studies  show  robot 
handlers  have  difficulties  comprehending  remote  environments  -  they  experience  what  domain  practitioners  often 
call  a  ‘soda  straw’.  This  illustrates  the  keyhole  effect  in  Human  Robot  Interaction,  a  CSE  phenomena  studied  in  the 
context  of  large  virtual  data  space  interfaces  and  the  current  research  seeks  to  reduce  this  effect.  A  simulation  for 
human-robot  coordinated  search  and  rescue  was  created  based  on  WTC  response  experiences.  Pilot  studies  showed 
traditional  performance  measures  to  be  inadequate  in  analyzing  control  and  exploration  tasks  therefore  a  novel 
analysis  approach  based  on  fractal  path  tortuosity  was  developed.  New  interface  concepts  for  helping  remote 
observers  perceive  environmental  affordances  were  then  tested  using  the  simulation  environment  and  evaluation 
measures.  These  studies  look  to  concepts  based  on  Gibsonian  principles  to  reduce  keyhole  effects  in  control 
interfaces  to  enhance  remote  functional  presence  in  Human-Robot  Coordination. 


Voshell,  M.  G.,  Woods,  D.  D.,  Pme,  B.,  &  Fern,  L.  C.  (2007).  Coordination  Loops:  A  New  Unit  of  Analysis  for 
Distributed  Work.  In  In  Proceedings  of  K.Mosier  &  U.  Fischer  (eds.).  Proceedings  of  the  Eighth  International 
Conference  on  Naturalistic  Decision  Making. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Joint  Activity  Theory,  automated  systems,  teamwork,  coordination  loops. 

Summary  abstract: 

Designing  effective  coordination  into  domains  of  distributed  decision  making  and  decentralized  control  is  a  daunting 
joint  cognitive  systems  challenge.  In  order  to  support  such  coordination,  it  is  necessary  to  look  at  modeling  these 
systems  from  the  perspective  of  coordination  requirements.  We  propose  “coordination  loops”  as  a  model  that 
enables  us  to  expand  upon  measures  and  constructs  that  allow  specification  requirements  of  distributed  work  to 
cultivate  effective  decision  making  across  multiple  domains. 


Warwick,  W.  (2002).  IICS  IPT:  The  M&S  of  Decision  Making.  In  In  Proceedings  of  Integrated  Infantry  Combat 
System  IPT.  Quantico,  VA. 
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Summary  abstract: 

•Goal:  Improve  realism  in  CGFs 

-Generic  prototype  built  around  task-network  modeling  tools  (Micro  Saint,  IMPRINT) 
-Exploit  a  client-server  architecture  to  “embed”  behaviors  in  DIS  or  HLA-compliant  CGF 


Warwick,  W.  (2003).  Developing  Computational  Models  of  Naturalistic  Decision  Making.  In  In  Proceedings  of 
Illinois  Wesleyan  University's  Natural  Sciences  Colloquium. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

Overview 

•A  rock  and  a  hard  place 
•Theoretical  background 
•Our  approach 

-What  it  means  to  model  a  naturalistic  decision  computationally 

-What  it  actually  looks  like  in  practice 

-Reactions 

•Reckless  and  inflammatory  remarks 


Warwick,  W.  (2003).  Developing  Computational  Models  of  Naturalistic  Decision  Making.  In  In  Proceedings  of 
Behavior  Representation  in  Modeling  and  Simulation  Conference.  Scottsdale,  AZ. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

What  Does  It  Mean  To  Develop  a  Computational  Model  of  a  Naturalistic  Decision? 

-  Naturalistic  theories  of  decision  making  discount  rational  choice  models  of  decision  making  in  favor  of 
descriptive  models 

=  Emphasis  on  experience,  situation  assessment  and  diagnosis  rather  than  the  deliberate  analysis  of 
alternatives 

For  the  last  three  years  we  have  been  working  toward  a  computational  representation  of  Klein’s  model  of  the 
recognition-primed  decision  (RPD).  After  months  of  tweaking  and  refining  in  an  attempt  to  capture  theoretical 
nuances,  we  have  finally  begun  a  process  of  beta  testing.  So  far,  this  has  taken  the  form  of  a  software  release  to  a 
small,  in-house  team  of  modelers  who  are  now  applying  our  representation  of  the  RPD  to  their  own  human 
performance  models.  Although  our  original  development  efforts  revolved  around  working  examples,  we  did  not 
anticipate  the  variety  of  questions  and  insights  that  would  arise  with  the  fresh  perspective  of  a  new  user  (our  in- 
house  modelers  were  not  part  of  the  original  development  team).  While  some  of  these  questions  were  the  result  of 
newly  discovered  bugs  and  usability  issues,  others  pointed  to  fundamental  assumptions  we  have  made  during  the 
course  of  this  work.  We  discuss  these  more  fundamental  issues.  In  particular,  we  describe  how  we  are  coming  to 
terms  with  some  of  the  limitations  inherent  in  a  computational  representation  of  a  naturalistic  decision  making 
process.  We  also  discuss  how  we  are  beginning  to  appreciate  the  extent  to  which  our  approach  to  modeling  decision 
making  differs  from  those  traditionally  used  in  the  development  of  Task-Network  models.  While  we  have 
occasionally  indulged  in  a  bit  of  sloganeering  in  past  discussions  of  our  work,  we  now  believe  we  have  identified 
some  genuine  methodological  differences  on  which  to  recommend  this  “naturalistic”  approach. 


Warwick,  W.  (2003).  Untangling  Computational  Intuitions  About  Naturalistic  Decision  Making  In  In  Proceedings  of 
Workshop  on  Cognitive  Systems:  Human  cognitive  models  in  system  design.  Hosted  by  Sandia  National 
Laboratories  and  the  University  of  New  Mexico.  Sante  Fe,  NM. 
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Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

Overview 
The  old  tangle 
Our  approach 

What  it  means  to  model  a  naturalistic  decision  computationally  in  general 
What  it  actually  looks  like  in  practice 
A  new  tangle 


Warwick,  W.  (2004).  Building  a  Cognitive  Menu  within  a  Cgf.  In  In  Proceedings  of  2004  Workshop  on  Cognitive 
systems:  Human  cognitive  models  in  system  design.  Hosted  by  Sandia  National  Laboratories  and  the  University  of 
New  Mexico.  Sante  Fe,  NM. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

The  Problem 

•The  behavioral  realism  of  the  synthetic  entities  that  populate  CGFs  is  lacking 
-rigid,  brittle  and  buggy 

-impacts  the  use  of  M&S  in  both  analysis  and  training 

-divorced  from  theory 

•No  easy  avenue  for  remediation 

-no  one  wants  you  to  mess  with  their  V&V’d  library  and  no  one  wants  to  start  over 
-lots  of  one-off  efforts  to  “hook”  individual  behaviors 


Warwick.  W.  (2009).  Chapter  24.  Understanding  the  Significance  of  Ada  Research.  In  P.  McDermott  and  L. 
Allender  (Eds.),  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  IV. 
Communicating  Information  across  the  Team:  48hrBooks  by  Alion  Science  &  Technology  Corporation  under 
contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and 
Engineering  Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative 
Technology  Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the 
Soldier  Decision  Maker.  July  22-23,  2009.  1100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003. 
http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

Over  the  nine-year  course'  of  the  ADA,  the  Army  has  undertaken  a  variety  oftechnological,  organizational,  and 
institutional  changes.  At  the  same  time,  the  Army  has  been  engaged  in  two  long-term  unconventional  conflicts.  The 
Army's  transition  to  a  digital  battlefield  conducted  under  the  pressures  of  asymmetric  warfare  has  led  to  significant 
transformations  in  tactics,  equipment,  a,nd  leadership.  This  transition  was  characterized  by  accelerated  technological 
transformations  which  increased  the  number  of  sensors  on  the  battlefield,  expanded  use  of  unmanned  and 
autonomous  assets,  and  leveraged  improvements  in  weapons  lethality,  responsiveness,  and  accuracy,  while 
dramatically  increasing  distance  between  echelons.  The  cumulative  effect  of  these  conditions  resulted  in  more 
decentralization  of  command  with  key  decision-makingtasks  pushed  to  lower  and  lower  echelons.  Combat  in  close 
quarters  increased  the  speed  of  the  "kill  chain"  and  led  to  a  commensurate  shortening  of  the  orient-observe-decide- 
act  loop.  The  operational  environment  was  further  complicated  by  increased  interdependence  with  other  services, 
reliance  on  Multi-National  Forces,  and  the  demand  to  operate  across  different  dimensions,  including  political, 
informational,  economic,  and  social.  At  the  tactical  level  of  combat,  daily  operations  decisively  shifted  from 
deliberate  to  more  adaptive  planning. 
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On  the  surface,  ADA  research  could  be  viewed  as  a  reaction  to  these  new  challenges;  Indeed,  much  of  the  success  of 
the  ADA  can  be  attributed  to  the  commitment  ADA  researchers  made  to  maintaining  operational  relevance.  But  the 
basic  research  ofthe  ADA  should  not  be  viewed  solely  through  the  lens  of  today's  circumstances.  The  science  of  the 
ADA  is  much  more  than  a  reaction  to  current  technology  trends  and  Army  contingencies  but,  rather,  rests  in  the 
development  of  a  deeper  understanding  of  the  invariant  challenges  of  warfare. 


Warwick,  W.  (2009).  Comparing  the  Comparisons.  In  Proceedings  ofthe  18th  Conference  on  Behavior 
Representation  in  Modeling  and  Simulation  (BRIMS),  Sundance,  UT,  31  March  -  2  April  2009. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Model  comparison  is  becoming  an  increasingly  common  method  in  computational  cognitive  modeling.  The 
methodology  is  seemingly  straightforward:  model  comparisons  invite  the  independent  development  of  distinct 
computational  approaches  to  simulate  human  performance  on  a  well-defined  task.  Typically,  the  benchmarks  of  the 
comparison  are  goodness-of-fit  measures  to  human  data  that  are  calculated  for  the  various  models.  Although  the 
quantitative  measures  might  suggest  that  model  comparisons  produce  “winners,”  the  real  focus  of  model  comparison 
is,  or  at  least  should  be,  on  understanding  in  some  detail  how  the  different  modeling  “architectures”  have  been 
applied  to  the  common  task.  And  in  this  respect,  the  seemingly  straightforward  method  of  model  comparison 
becomes  more  complicated. 


Warwick.  W.,  Allender,  L.,  &  Yen,  J.  (2009).  Editors’  Introduction  to  the  Special  Issue  on  Developing  and 
Understanding  Computational  Models  of  Macrocognition.,  Special  Section  ofthe  Journal  of  Cognitive  Engineering 
and  Decision  Making:  Human  Factors  and  Ergonomics  Society  (HFES). 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Newell  and  Simon  (1976)  once  famously  characterized  computer  science  as  an  empirical  inquiry — not  merely  an 
engineering  discipline  or  a  branch  of  applied  mathematics  but  a  science  concerned  with  the  discovery  of  the 
“essential  nature”  of  symbol  systems.  Even  more  famously,  they  presented  the  physical  symbol  system  and  heuristic 
search  hypotheses  as  candidate  laws  of  the  qualitative  structure  of  intelligent  systems.  Their  claims  were  not  just 
about  computers  but  about  the  nature  of  intelligence  in  general — namely,  that  the  realization  of  a  physical  symbol 
system  provides  both  necessary  and  sufficient  conditions  for  intelligence  and  that  the  hallmark  of  such  intelligence 
is  the  efficient  search  and  testing  of  solutions  within  a  problem  space.  It  would  be  hard  to  overstate  the  impact  of 
these  hypotheses  within  the  fields  of  artificial  intelligence  and  cognitive  science,  as  researchers  since  then  have 
either  elaborated  these  hypotheses  or  reacted  to  them. 


Warwick,  W.,  Archer,  R.,  Hamilton,  A.,  Matessa,  M.,  Santamaria,  A.,  Chong,  R.  S.,  Allender,  L.,  &  Kelley,  T. 
(2008).  Integrating  Architectures:  Dovetailing  Task  Network  and  Cognitive  Models.  In  In  Proceedings  of  Behavior 
Representation  in  Modeling  and  Simulation  (BRIMS  '08). 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Task  Network  Modeling;  Cognitive  Modeling;  C3TRACE;  ACT-R 

Summary  abstract: 
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This  paper  describes  our  efforts  to  develop  the  Human  Behavior  Architecture  (HBA),  an  integration  of  task  network 
and  cognitive  modeling  tools.  We  begin  by  describing  the  two  component  modeling  tools  we  have  integrated: 
C3TRACE,  a  task  network  modeling  tool  and  the  ACT-R  cognitive  architecture.  Next,  we  describe  the  current  state 
of  the  integration,  highlighting  practical  aspects.  We  then  turn  to  a  discussion  of  conceptual  issues  that  have  surfaced 
as  we  put  integration  into  practice.  Finally,  we  discuss  the  benefit  we  see  in  this  integration. 


Warwick,  W.,  &  Archer,  S.  (2001).  Incorporating  Aspects  of  Human  Decision  Making  in  Task  Network  Simulation 
Tools.  In  In  Proceedings  of  2001  NATO  RTA  Conference  on  Future  Modeling  and  Simulation  Challenges.  Breda, 
Breda,  Netherlands,  12-14  November  2001,  and  published  in  RTO-MP-073. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

In  this  paper  we  describe  three  ongoing  projects  intended  to  improve  the  representation  of  human  decision-making 
in  military  simulations.  Each  project  addresses  a  different  aspect  of  decision  making.  The  first  project  extends  the 
functionality  of  the  Improved  Performance  Research  Integration  Tool  (IMPRINT)  by  allowing  the  user  to  create  a 
detailed  model  of  a  goal-oriented  human  agent.  A  simulation  running  in  IMPRINT  predicts  what  the  human  is  likely 
to  do  based  on  the  currently  relevant  goals  and  the  status  of  other  parallel  simulations.  The  focus  of  the  second 
project  is  to  predict  the  likelihood  of  a  particular  decision  being  made  successfully  given  the  quality  of  information 
available  at  the  time  the  decision  is  made.  The  underlying  idea  is  to  use  a  task-network  model  to  represent  who  knew 
what  and  when.  In  our  third  project,  we  are  working  to  represent  the  human  decision-making  process  in  time- 
pressured,  stressful  situations.  We  have  turned  to  Klein’s  theory  of  the  Recognition-Primed  Decision  (RPD)  as  a 
model  of  what  people  actually  do  in  such  situations.  RPD  theory  differs  from  traditional,  analytical  theories  of 
decision  making  insofar  as  the  emphasis  lies  on  situation  assessment  rather  than  the  comparison  of  options  and  thus 
poses  a  novel  set  of  computational  challenges. 


Warwick,  W.,  Brockett,  C.,  Mcllwaine,  S.,  Hutton,  R.,  &  Hahn,  B.  (2002).  Incorporating  Models  of  Recognition- 
Primed  Decision  Making  in  Computer  Generated  Forces.  In  In  Proceedings  of  Simulation  Interoperability 
Workshop.  Orlando,  FL,  SISO. 

Key  Words: 
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Modeling  and  Simulation,  Naturalistic  Decision  Making,  Computer  Generated  Forces 

Summary  abstract: 

In  this  paper  we  describe  a  six  month  effort  undertaken  by  Micro  Analysis  and  Design,  Inc.,  and  Klein  Associates, 
Inc.,  to  investigate  a  potential  improvement  to  the  representation  of  human  decision  making  within  computer 
generated  forces  (CGFs).  Like  other  efforts,  our  work  was  motivated  by  the  need  to  improve  behavioral  realism  in 
CGFs  and  to  bring  theoretically  grounded  models  of  human  performance  to  military  simulations.  Unlike  other 
efforts,  however,  our  approach  to  decision  making  was  not  rooted  in  the  application  of  an  integrative  cognitive 
architecture.  Instead  we  focused  on  individual  “decision  points”  in  the  CGF  and  considered  how  we  might  supplant 
the  routines  at  those  points  with  “naturalistic”  models  of  decision  making  intended  to  reflect  what  humans  actually 
do  in  real  situations. 

Originally  the  main  thrust  of  our  effort  was  to  investigate  the  extent  to  which  we  could  implement  a  naturalistic 
representation  within  the  OneSAF  Test  Bed  environment.  It  soon  became  clear,  however,  that  a  significant 
additional  effort  was  needed  just  to  determine  whether  our  approach  was,  in  fact,  leading  to  observable  or  interesting 
changes  in  entity  performance.  We  briefly  describe  below  the  “decision  points”  we  identified  in  the  OneSAF  Test 
Bed  and  how  our  naturalistic  approach  is  realized  within  the  OneSAF  Test  Bed  environment.  Next,  we  turn  to  a 
more  extended  discussion  of  the  approach  we  took  to  determine  whether  our  efforts  were  yielding  interesting 
changes  in  decision  making  performance.  In  particular,  we  describe  a  pseudoSAF  model  we  developed  to  help  us 
isolate  and  understand  individual  behaviors  in  the  OneSAF  Test  Bed.  We  then  describe  how  our  pseudoSAF  lays  the 
foundation  for  a  more  generic  test  architecture  that  will  provide  a  controlled  environment  in  which  to  compare  new 
models  of  human  performance  against  extant  representations. 


287 


An  Annotated  Bibliography  of  Studies  and  Reports  Produced  by  the  Advanced  Decision  Architectures  Consortium 
of  the  Collaborative  Technology  Alliance  from  2001  to  2010 


Warwick,  W.,  &  Fleetwood,  M.  (2006).  A  Bad  Hempel  Day:  The  Decoupling  of  Explanation  and  Prediction  in 
Computational  Cognitive  Modeling.  In  In  Proceedings  of  Fall  2006  Simulation  Interoperability  Workshop.  Orlando, 
FL.  SISO. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Cognitive  Modeling,  Prediction,  Explanation 

Summary  abstract: 

The  philosopher  of  science  Gustav  Hempel  famously  argued  for  the  symmetry  of  prediction  and  explanation  in 
scientific  practice.  Pointing  to  parallels  in  their  logical  structure,  Hempel  maintained  that  any  adequate  scientific 
explanation  could  engender  a  prediction,  and  vice  versa.  This  symmetry  is  evident  in  the  practice  of  computational 
cognitive  modeling.  Indeed,  much  of  the  motivation  for  computational  cognitive  modeling  follows  from  the  belief 
that  process-level  representations  (i.e.,  the  implementation  of  explanatory  accounts  of  cognition)  will  lead  to  better 
predictions  of  cognitive  performance  and,  conversely,  that  such  predictions  can  be  used  to  confirm  or  refute  the 
accounts  of  cognition  so-modeled.  In  this  paper  we  present  the  results  of  a  case  study  in  which  we  undertook  three 
very  different  modeling  approaches  to  a  well-known  categorization  task  in  experimental  psychology.  The 
approaches  ran  the  gamut  in  terms  of  the  commitments  they  made  to  the  nature  of  the  cognitive  processes 
underlying  the  task.  At  the  end  of  the  day,  however,  each  modeling  approach  yielded  good  and,  moreover, 
essentially  indistinguishable  fits  to  the  human  performance  data.  So,  what  does  it  say  about  the  relationship  between 
prediction  and  explanation  in  cognitive  modeling  when  three  qualitatively  different  approaches  yield  quantitatively 
similar  results?  Several  responses  are  possible.  We  canvass  two  such  responses  before  discussing  what  we  consider 
to  be  a  more  basic  concern  about  computational  approaches  to  cognitive  modeling. 


Warwick,  W.,  &  Hayes,  P.  (2003).  Developing  Computational  Models  of  Naturalistic  Decision  Making: 
Methodologies  and  Perspectives.  In  In  Proceedings  of  Collaborative  Technology  Alliances  Conference  2003. 

Science  and  Technology  Innovations  for  the  Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland 
University  College,  Adelphi,  Maryland:  U.S.  Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

In  this  paper  we  discuss  several  methodological  issues  that  have  emerged  during  our  efforts  to  develop  a 
computational  model  of  a  "naturalistic"  decision  making  process.  Our  original  goal  was  to  improve  the  realism  of 
computer  generated  forces  by  providing  a  framework  to  represent  recognitional  decisions.  Along  the  way,  we  have 
come  to  appreciate  the  interplay  between  the  decision  maker's  explicit  knowledge  (e.g.,  knowing  which  course  of 
action  is  appropriate  for  a  given  set  of  cues)  and  implicit  knowledge  (e.g.,  knowing  how  much  significance  to  attach 
to  a  particular  cue  in  a  given  situation,  how  finely  to  resolve  that  cue,  how  much  uncertainty  to  tolerate  when 
assessing  a  situation,  etc.).  We  have  also  come  to  recognize  three  things  about  modeling  human  decision  making.: 
First,  realistic  human  decision  making  cannot  be  represented  solely  in  terms  of  expert  knowledge  and  that  we  need 
to  think  as  much  about  underlying  process  as  we  to  overt  procedure.  Second,  the  representation  of  situated  decision 
making  behavior  can  benefit  from  the  insights  of  both  the  cognitive  modeling  and  artificial  intelligence  communities 
despite  the  fact  that  both  traditions  are  often  discounted  by  the  Naturalistic  Decision  Making*  community.  Third, 
there  are  basic  ontological  and  epistemological  assumptions  that  underlie  any  representation  of  human  behavior  that 
are  taken  for  granted  all  too  often.  Not  surprisingly,  the  failure  to  account  for  these  assumptions  can  lead  to  some 
deep  confusions.  We  discuss  each  of  these  issues  below  in  the  context  of  our  own  work  with  an  eye  toward 
discovering  a  methodologically  clear  route  to  an  improved  understanding  of  human  decision  making,  both 
conceptual  and  computational. 


Warwick,  W.,  &  Hutchins,  S.  (2004).  Initial  Comparisons  between  a  Naturalistic  Model  of  Decision  Making  and 
Human  Performance  Data.  In  In  Proceedings  of  13th  Conference  on  Behavior  Representation  in  Modeling  and 
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Simulation.  Arlington,  VA,  SISO.  pp.  73-78. 


Key  Words: 
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Summary  abstract: 

In  our  past  papers  we  have  described  the  mechanics  of  a  naturalistic  model  of  decision  making  and  we  have  outlined 
its  performance  in  qualitative  terms.  In  this  paper  we  describe  our  initial  attempts  to  compare  the  performance  of  our 
naturalistic  model  of  decision  making  against  human  decision  making  performance.  The  human  performance  data 
were  collected  from  an  experiment  independently  conducted  to  determine  how  a  graphical  user  interface  might  be 
best  designed  to  support  optimal  decisions  in  a  resource  allocation  task.  Our  goal  was  to  model  just  one  of  the 
experimental  conditions  and  see  whether  we  could  arrive  at  the  same  the  allocation  results  using  the  highlevel 
representations  supported  by  the  naturalistic  model.  The  initial  comparisons  have  been  promising,  but  they  also  raise 
interesting  questions.  We  describe  below  both  the  experimental  design  and  the  naturalistic  model  that  simulates  that 
design.  We  then  present  the  results  of  the  quantitative  comparison.  Finally,  we  discuss  some  of  the  larger  issues  we 
see  emerging  from  this  comparison. 


Warwick,  W.,  &  Hutton,  R.  J.  B.  (2007).  Chapter  20.  Computational  and  Theoretical  Perspectives  on  Recognition 
Primed  Decision  Making.  In  R.  Hoffman  (Ed.),  Naturalistic  Decision  Making  (Vol.  6).  Mahwah,  NJ.:  Lawrence 
Erlbaum  Associates,  Inc. 
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Summary  abstract: 

A  theme  of  chapters  in  this  volume  is  the  question  of  how  experts  cope  with  challenging,  rare,  or  unusual  problems, 
whether  their  expertise  can  transfer  to  novel  situations,  and  how  expertise  can  accommodate  dynamic  and  uncertain 
situations.  The  goal  is  to  understand  how  expertise  is  manifested  when  experts  have  to  work  “out  of  context.”  In  this 
chapter,  we  take  the  theme  at  a  90-degree  angle,  as  we  try  to  understand  expertise  not  just  out  of  context,  but  out  of 
body.  Rather  than  discuss  expertise  stripped  from  familiar  circumstances,  we  discuss  expertise  stripped  from  the 
expert.  The  discussion  focuses  on  our  attempts  to  develop  a  computational  model  of  the  recognition-primed  decision 
(RPD)  (Klein,  1993,  1998). 

Our  sense  of  disembodiment  follows  from  the  fact  that  we’re  trying  to  identify  and  computationally  implement  some 
of  the  invariant  aspects  of  a  Naturalistic  Decision-Making  (NDM)  process.  The  ultimate  goal  of  the  computational 
work  is  not  to  develop  a  representation  of  this  person  making  that  decision,  but  rather,  to  develop  a  general 
architecture  that  can  be  applied  to  the  representation  of  a  variety  of  decision  makers  in  a  variety  of  real-world 
contexts.  In  turn,  these  representations  should  allow  us  to  simulate  decision  making  more  realistically  and  in  a 
manner  with  greater  psychological  plausibility  than  the  probabilistic  and  rule-based  representations  in  common  use 
today  in  computational  modeling. 

From  a  theoretical  perspective,  one  goal  is  to  address  the  question  of  whether  it  is  even  possible  to  develop  a 
computational  model  of  an  NDM  process.  Indeed,  some  might  argue  that  naturalistic  models  of  decision  making  are 
interesting  precisely  because  their  noncomputational  nature  seems  to  not  fit  with  microscale  representations  (Klein, 
Ross,  et  ah,  2003),  and  that  it  simply  makes  no  sense  to  try  to  coerce  macrocognitive  phenomena  into  a 
computational  framework.  Second,  if  we  assume  it  is  possible  to  model  such  processes  computationally,  we  must  fix 
the  level  of  abstraction  at  which  we  claim  correspondence  between  the  theoretical  and  computational  model  and  we 
must  clearly  delineate  the  sense  in  which  the  resulting  computational  models  are  “naturalistic.”  As  a  third  goal,  we 
must  establish  a  reciprocal  relationship  between  the  two  models,  not  just  one  where  the  theoretical  model  informs 
the  computational,  but  rather,  where  the  computational  model  helps  us  explore  aspects  of  the  theoretical  description 
that  might  remain  underspecified. 

In  this  chapter,  we  report  on  our  progress  toward  these  three  goals.  We  begin  with  a  brief  overview  of  the  theory  and 
the  computational  model.  Next,  we  turn  to  the  question  of  how  well  the  computational  model  represents  the  theory. 
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We  describe  several  successive  attempts  we  have  made  to  model  one  of  the  several  “by-products”  of  recognition 
posited  by  the  RPD  theory.  Each  iteration  of  this  process  has  been  instructive,  in  terms  of  both  raising  theoretical 
questions  and  identifying  limitations  in  the  computational  representation.  Finally,  we  describe  the  implications  we 
see  in  this  work  for  understanding  the  relation  between  the  theory  and  computational  model.  This  relationship  has 
turned  out  to  be  more  subtle  than  we  expected,  and  it  has  caused  us  to  rethink  some  of  our  most  basic  assumptions 
about  the  limitations  of  both  the  theory  and  the  computational  model. 


Warwick,  W.,  Mcllwaine,  S.,  Hutton,  R.,  &  McDermott,  P.  (2002).  Developing  Computational  Models  of 
Recognition-Primed  Decision  Making.  In  In  Proceedings  of  1 1th  Conference  on  Computer  Generated  Forces  and 
Behavior  Representations.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Decision  Making,  Recognition-Primed,  Task-Network  Modeling 

Summary  abstract: 

In  this  paper  we  describe  our  ongoing  work  to  develop  a  computational  representation  of  Klein’s  model  of 
Recognition-Primed  Decision  making  (RPD).  The  RPD  model  differs  from  traditional,  analytical  models  of  decision 
making  insofar  as  RPD  emphasizes  situation  assessment  rather  than  the  comparison  of  options.  Like  many  efforts, 
our  research  is  motivated  by  the  need  for  improved  realism  in  the  behavior  of  Computer  Generated  Forces,  and  in 
particular,  the  need  for  better  representations  of  human  decision  making.  Unlike  other  efforts,  however,  our  attempt 
to  model  RPD  is  not  part  of  a  more  general  model  of  cognition.  We  describe  here  the  extent  to  which  we  have  been 
able  to  model  RPD  apart  from  other  aspects  of  cognition  and  we  present  the  resulting  RPD-specific  architecture.  We 
also  point  to  the  test  bed  environment  we  used  for  proof-of-concept  and  our  current  efforts  to  scale-up  to  a  more 
complex  decision  making  environment.  Finally,  we  discuss  our  vision  of  a  computational  model  of  RPD  as  one  tool 
in  a  suite  of  task  networked  based  models  of  human  decision  making  and  the  potential  advantages  that  follow  from 
this  approach. 


Warwick,  W.,  &  Santamaria,  A.  (2006).  Giving  up  Vindication  in  Favor  of  Application:  Developing  Cognitively- 
Inspired  Widgets  for  Human  Performance  Modeling  Tools.  In  In  Proceedings  of  7th  International  Conference  on 
Cognitive  Modeling.  Trieste,  IT.  Edizioni  Goliardiche. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Computational  models  of  cognition  are  most  often  developed  to  explore  or  vindicate  particular  theoretical  views  in 
psychology.  The  computer  provides  a  ready  environment  in  which  to  develop  models  and  generate  quantitative 
predictions  about  cognitive  performance  which,  in  turn,  can  be  compared  against  actual  human  performance. 
Validating  the  model  with  a  good  fit  vindicates  the  theoretical  view  the  model  implements.  The  resulting  view  of  the 
cognitive  modeling  enterprise  is  decidedly  hypothetico  deductive.  For  all  its  familiar  appeal,  however,  this  is  not  the 
only  reason  to  develop  a  cognitive  model.  In  this  paper,  we  describe  our  efforts  to  develop  computational  models  of 
decision  making  that  can  be  applied  within  existing  human  performance  modeling  tools.  Although  our  efforts  are 
inspired  theoretically,  the  goal  is  not  to  vindicate  this  or  that  theoretical  view  of  cognition  but,  rather,  to  engender 
sufficiently  human-like  behavior  in  a  modeling  environment  where  it  is  otherwise  lacking.  We  begin  by  briefly 
describing  the  computational  nuts  and  bolts  we’ve  developed  for  representing  ‘naturalistic’  decisions  in  a  task 
network  modeling  environment.  We  then  point  to  results  from  several  applications  to  demonstrate  the  flexibility  of 
our  approach  and  the  extent  to  which  it  does  really  produce  human-like  behaviors,  both  quantitatively  and 
qualitatively.  We  conclude  by  discussing  the  alternative,  more  instrumentalist  view  of  the  cognitive  modeling 
enteiprise  that  results  when  efforts  are  focused  on  the  development  of  applications  rather  than  architectures. 


Warwick,  W.,  &  Santamaria,  A.  (2009).  Chapter  15.  A  Computational  Model  of  Naturalistic  Decision  Making  and 
the  Science  of  Simulation.  In  P.  McDermott  and  L.  Allender  (Eds.),  Advanced  Decision  Architectures  for  the 
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Warfighter:  Foundations  and  Technology.  Section  III.  Acting  on  Battlefield  Information:  48hrBooks  by  Alion 
Science  &  Technology  Corporation  under  contract  to  the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving 
Ground,  MD  21005,  Human  Research  and  Engineering  Directorate.  Proceedings  from  the  Advanced  Decision 
Architectures  (ADA)  group  of  the  Collaborative  Technology  Aliance  (CTA)  Final  Research  Management  Board 
Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision  Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue 
SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr-webtest.alionscience.com/AdaCta/Agenda.aspx. 

Key  Words: 

Collaborative  Technology  Alliance,  CTA,  Advanced  Decision  Architectures,  ADA 
Summary  abstract: 

With  its  roots  in  experimental  psychology,  computational  cognitive  modeling  is  most  often  seen  as  an  exercise  in 
theory  testing.  This  view  holds  that  theories  of  cognition  are  formalized  as  executable  software  programs;  these 
programs  are  exercised  as  simulations  of  laboratory  tasks  to  generate  predictions  that  are  then  compared  to  actual 
human  performance  on  those  same  tasks.  Good  fits  validate  good  theories. 

Although  this  view  has  received  its  share  of  scrutiny,  most  of  the  concern  has  focused  on  the  finer  points  ofthe 
method  (e.g.,  the  quality  of  the  data  used  for  validation)  while  the  most  basic  assumption  of  the  entire  enterprise-that 
theories  of  cognition  can  be  expressed  directly  as  computational  models-remains  unquestioned.  Of  course,  some 
would  argue  that  the  relationship  between  cognitive  theory  and  computer  model  is  immediate.  Indeed,  Simon  (1996) 
wrote,  "  ...  if  computers  are  organized  somewhat  in  the  image  of  man,  then  the  computer  becomes  the  obvious 
device  for  exploring  the  consequences  of  alternative  organizational  assumptions  for  human  behavior  ..."  (pg.  21; 
emphasis  added).  Similarly,  Newe"  (1990)  proposed  that  the  layered  abstractions  of  computing  systems  provide  a 
useful  approach  for  describing  and  understanding  "knowledge  systems."  Even  those  who  espouse  a  connectionist 
view  of  the  human  cognitive  architecture  (e.g.,  McClelland  and  Rumelhart,  1989)  find  comfort  in  computational 
models  once  the  right  functional  units  have  been  abstracted  from  the  neurology.  While  cognitive  modelers  might 
debate  the  merits  of  symbolic  versus  distributed  processing,  they  do  not  argue  about  the  adequacy  of  productions 
systems  or  neural  networks  as  computational  models  ofthose  processes. 

The  concern  here  is  that  the  seemingly  direct  relationships  between  the  computational  models  and  the  cognitive 
theories  they  purport  to  represent  are  dependent  on  the  theories  themselves  and  not  on  any  general  understanding  of 
the  role  of  computer  simulation  in  cognitive  science.  What  happens  when  we  consider  a  theory  of  cognition  that 
doesn't  happen  to  map  easily  or  directly  onto  a  computational  architecture?  What  general  guidance  is  there  to  decide 
what  needs  to  be  included  in  a  cognitive  model?  Is  there  a  theoretically  neutral  account  of  what  constitutes  a  proper 
simulation?  These  are  not  idle  concerns.  In  fact,  nowhere  are  these  concerns  more  present  than  when  we  look  at  the 
role  of  computer  simulation  in  representing  the  macrocognitive  theories  of  cognition  (e.g.,  Cacciabue  and  Hollnagel, 
1995).  These  theories  are,  in  large  part,  predicated  on  a  rejection  ofthe  view  of  cognition  as  information  processing 
and  they  are  typically  couched  in  descriptive  terms  far  above  the  level  of  detail  where  a  "micro-level"  process, 
neural  or  otherwise,  might  be  implemented.  Moreover,  research  in  macrocognition  is  typically  conducted  "in  the 
wild,"  beyond  any  laboratory  control  that  might  lend  itselfto  computational  representation.  Simulating  and 
comparing  performance  on  a  problem  solving  task  like  the  Towers  of  Hanoi  is  far  easier  than  trying  to  capture  the 
vagaries  of  expertise  as  it  is  applied  in  a  dynamic  and  perhaps  ill-structured  real  world  domain. 

In  this  chapter  we  report  on  our  attempt  to  come  to  grips  with  these  issues  with  the  development  of  a  computational 
model  of  the  recognition-  primed  decision  (RPD),  a  key  function  postulated  by  theories  of  macrocognition.  We 
begin  by  describing  the  intended  correspondence  between  the  computational  and  theoretical  model.  We  then  briefly 
describe  some  tasks  to  which  we've  applied  the  model  and  the  extent  to  which  model  predictions  have  fit  human 
performance  data.  Though  these  applications  provide  a  good  degree  of  validation  for  the  model,  we  conclude  by 
discussing  some  of  the  broader  implications  we  see  in  this  kind  of  work  for  understanding  cognitive  modeling  and 
simulation  as  a  scientific  endeavor. 


Weil,  S.  A.,  Tinapple,  D.,  &  Woods,  D.  D.  (2004).  New  Approaches  to  Overcoming  E-Mail  Overload.  In  In 
Proceedings  of  48th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  New  Orleans,  LA. 
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Summary  abstract: 
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As  e-mail  has  become  the  preferred  medium  for  communication,  the  inbox  and  mail  folders  become  one  hub  for 
organizing  activities  and  schedule.  The  combination  of  a  natural  rise  in  message  volume  and  the  large  amounts  of 
unsolicited  bulk  messages  (spam)  have  led  some  to  suggest  that  the  usefulness  of  e-mail  is  at  an  end;  users  feel 
overwhelmed  by  the  sheer  numbers  of  individual  messages  and  the  effort  to  manage  the  inbox.  This  paper  frames 
the  issue  of  e-mail/message  overload  as  a  specific  example  of  data  overload  and  uses  previous  results  to  suggest 
three  design  concepts  -  cognitive  buoyancy,  e-mail  constellations,  and  the  intelligent  subject  line  -  for  use  in  e-mail. 


Williges,  R.  C.  (2005).  An  Invited  Presentation  Based  on  Project  Activities  Titled  Interactive  Applied  Experimental 
Design  and  Analysis  Tools  for  Engineering  Psychology.  In  In  Proceedings  of  Division  21  President's  Invited  Paper 
presented  at  the  2005  American  Psychological  Association  Convention,  Washington,  D.C. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 

Summary  abstract: 

1 .  Engineering  Psychology  Methods 

-  Research,  Design,  and  Evaluation  Methods 

-  Applied  Experimental  Design  and  Analysis 

2.  Example  of  an  Online  Tool 

-  Computer-Based  Reference  Material 

-  Interactive  Reference  Material  Topics 

3.  Conclusions 

-  Characteristics  of  Effective  Online  Tools 

-  Online  Toolkit  Implication 

4.  Prototype  Interactive  Tool  Demonstration 


Williges,  R.  C.,  &  Middlebrooks,  S.  (2007).  Experimental  Design  Analysis  of  U.S.  Army  Command  and  Control 
Systems  with  the  CADRE  Tool.  In  In  Proceedings  of  25th  Army  Science  Conference,  Orlando,  FL. 

Key  Words: 
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Chi-Square  Test  of  Independence  (2x2  Contingency  Table) 

Chi-Square  Test  of  Independence  (RxC  Contingency  Table) 

Chi-Square  Test  of  Independence  (Two  Additive  2x2  Partitions) 

McNemar  Change  Test 

Cochran  Q  Test 

Kolmogorov-Smirnov  Tests 

Kruskal-Wallis  One-Way  ANOVA 

Wilcoxon  Signed  Ranks  Test 

Friedman  Two-Way  ANOVA 

One-Factor,  Between-Subjects  ANOVA 

Two-Factor,  Between-Subjects  ANOVA 

Planned  Comparisons 

Unplanned  Comparisons 

Analysis  of  Interactions 

One-Factor,  Within-Subjects  ANOVA 

Two-Factor,  Within-Subjects  ANOVA 

Geisser-Greenhouse  and  Huynh-Feldt  Corrections 
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Testing  Order  Effects  in  Balanced  Latin  Squares 

Within-Subjects  and  Between-Subjects 

Design  Comparison 

Two-Way,  Mixed-Factors  ANOVA 

Complete  Hierarchical  Between-Subjects  Design 

Partial  Hierarchical  Between-Subjects  Design 

Simple  Blocking  of  2k  Within-Subjects  Design 

Complex  Blocking  of  2k  Within-Subjects  Design 

One-Half  Replicate  of  24  Between-Subjects  Design 

Linear  Correlation  Coefficient 

Alternative  Linear  Correlations 

Simple  Linear  Regression 

Analysis  of  Covariance  (ANCOVA) 

One-Way,  Between-Subjects  Design 
Multiple  Linear  Regression 
Best  Regression  Equation 
Polynomial  Regression 

Orthogonal,  Between-Subjects,  Central-Composite  Design 
Blocked,  Within-Subjects,  Central-Composite  Design 

Summary  abstract: 

Previous  work  has  involved  experimental  design  interrogation  of  a  series  of  computer  models  of  human  task  and 
workload  performance  in  military  command  and  control  (C2)  systems.  These  models,  called  Computer  modeling  Of 
Human  Operator  System  Tasks  (CoHOST)  (Middlebrooks  et  al.,  1999),  have  shown  a  dramatic  need  for  some  tool 
to  assist  experimental  designers  in  efforts  to  establish  effective  experiments.  Specific  needs  include  the  collection  of 
data  in  a  manner  that  minimizes  the  amount  of  data  required  while  maximizing  the  effectiveness  and  power  of  the 
data.  A  desktop  tool  called  Computer-Aided  Design  Reference  for  Experiments  (CADRE)  was  developed  for  this 
purpose.  The  CADRE  tool  includes  over  850  pages  of  reference  material  covering  25  topics  that  are  divided  into 
five  major  sections  including  an  introduction  to  experimental  design,  supplemental  data  collection  design  and 
analysis,  basic  analysis  of  variance  (ANOVA)  designs,  advanced  ANOVA  designs,  and  empirical  model  building.  In 
addition,  the  CADRE  tool  contains  over  200  pages  explaining  39  examples  of  statistical  analyses  covered  in  the 
reference  material  and  is  hyperlinked  to  Version  9.1.3  of  the  SAS  statistical  analysis  package.  The  CADRE  tool  can 
be  used  for  choosing  experimental  design  procedures  to  interrogate  and  build  empirical  models  in  support  of 
complex  studies  such  as  future  C2  systems  computer  simulations. 


Wojciechowski,  J.  Q.,  &  Archer,  S.  G.  (2002).  Human  Performance  Modeling  -  Art  or  Science?  In  In  Proceedings  of 
Military,  Government,  and  Aerospace  Simulation  (MGA  2002)  Conference.  Michael  Chinni,  (Ed.). 

Key  Words: 
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Human  Performance 

Summary  abstract: 

Human  performance  modeling  is  being  used  in  many  different  situations.  It  has  been  applied  early  in  system 
acquisition  in  order  to  help  influence  the  design  so  that  the  human  functions  more  effectively  as  a  system 
component.  It  has  also  been  applied  in  operational  testing,  in  order  to  provide  guidance  on  how  to  best  apply  limited 
human-in-the-loop  testing  resources.  These  efforts  have  met  with  varying  degrees  of  success,  even  when  the  same 
tool  set  is  applied.  Additionally,  elaborate  models  and  simulations  have  been  built  at  great  cost  and  effort,  only  to  sit 
on  the  shelf  because  they  did  not  effectively  answer  a  relevant  question.  This  has  led  to  discussions  about  the  level 
to  which  building  a  good  human  performance  model  or  simulation  is  art  versus  science. 

Admittedly,  developing  predictive  human  performance  models  is  not  an  easy  task.  However,  the  human  is  one  of 
the  primary  contributors  to  the  variability  in  a  system,  and  human  performance  modeling  tools  allow  us  to  represent 
the  human’s  role  in  the  system  early  in  the  design  phase  so  that  we  can  develop  systems  that  are  more  usable, 
maintainable,  and  supportable  by  their  human  components.  Many  believe  that  developing  a  model  that  effectively 
represents  the  human  is  an  art  —  that  it  takes  years  of  practice  and  a  fair  amount  of  inherent  talent  to  build  good 
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human  performance  models.  Others  argue  that  it  is  an  engineering  exercise,  and  good  science  alone  determines  the 
predictiveness  of  the  final  result. 

In  this  paper  we  explore  those  issues  in  order  to  better  understand  the  key  to  creating  good  human  performance 
models  and  modelers.  We  present  several  examples  and  point  to  elements  of  the  models  that  determined  their 
success.  We  conclude  that  good  science  is  a  necessary  but  not  sufficient  element  of  good  human  performance 
modeling. 


Woods,  D.  (2005).  Generic  Support  Requirements  for  Cognitive  Work:  Laws  That  Govern  Cognitive  Work  in 
Action.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Also  Invited  talk.  IBM  Research  Center. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Research  in  Cognitive  Systems  Engineering  (CSE)  has  successfully  identified  basic  requirements  that  must  be  met  if 
new  technology  will  be  useful  to  practitioners  in  context.  Synthesizing  these  basic  requirements  or  support  functions 
is  part  of  a  process  of  debate  and  consolidation  of  the  foundations  of  the  field  after  25  years  of  productive  activity 
(Klein,  1999;  Endsley  et  al.,  2003;  Hollnagel  and  Woods,  2005).  This  work  takes  the  “Laws  that  Govern  Cognitive 
Work”  which  synthesize  basic  findings  aand  patterns  (Woods,  2002;  Hoffman  and  Woods,  2005)  and  provides  the 
next  step — a  set  of  basic  requirements  or  support  functions  for  design.  General  requirements  for  effective  support 
can  be  used  to  jump  start  individual  development  projects  in  any  domain.  Debating  how  to  achieve  these  support 
functions  helps  translate  the  insights  of  cognitive  work  analyses  into  tangible  new  uses  of  technological  possibilities. 


Woods,  D.,  &  Christoffersen,  K.  (2001).  Balancing  Practice-Centered  Research  and  Design.  In  M.D.  McNeese  and 
M.A.  Vidulich  (Eds.),  Cognitive  Systems  Engineering  in  Military  Aviation  Domains,  Pp.  121-136.  Wright-Patterson 
AFB,  OH:  Human  Systems  Information  Analysis  Center  (HSIAC). 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

Many  are  interested  in  R&D  at  the  intersection  of  people,  technology  and  work.  Developers  and  technologists  make 
claims  about  how  a  prospective  new  capability  or  new  system  development  project  will  impact  on  performance  in 
one  or  many  settings.  Sponsors  are  caught  up  in  the  sweeping  dreams  permitted  by  technology  unfettered  from  harsh 
contexts  of  use,  yet  they  fear  software  development  projects  that  fail  to  provide  useful  tools  or  that  create 
unanticipated  negative  effects.  Practitioners  and  observers  of  practitioners  at  work  note  repeated  forms  of  clumsiness 
in  the  technology  deployed  and  unanticipated  side  effects  of  change.  Researchers,  blinded  by  the  glare  of 
disciplinary  labels,  drastically  reduced  situations  to  fit  into  a  lab  one  variable  at  a  time  yet  claim  priority  in  the 
search  for  generic  regularities.  Human  factors  practitioners  and  usability  engineers  are  called  in  too  late  to  repair  the 
connection  between  systems  and  use.  Research  results  seem  irrelevant  to  design.  Design  seems  local  and  unique. 


Woods,  D.,  McNee,  S.,  Davis,  J.  W.,  Morison,  A.,  Maughan,  P.,  &  Christoffersen,  K.  (2005).  Event  Template 
Hierarchies  as  Means  for  Human- Automation  Collaboration  in  Security  Surveillance.  In  In  Proceedings  of  49th 
Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 
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Advances  in  remote  sensing  systems  provide  human  monitors  access  to  more  data.  The  current  challenge  is  to  help 
extract  relevant  patterns  and  direct  the  attention  of  human  monitors  and  human  problem  holders  to  the  changing 
security  picture  with  respect  to  acute  situations  (a  normal  market  scene  turns  into  an  ethnic  confrontation)  or  longer 
term  trends  (seeing  new  patterns  of  ‘typical’  behavior  to  avoid  false  alarms).  Security  surveillance  monitoring  can  be 
advanced  through  new  event  recognition  capability  of  autonomous  monitors  and  by  effectively  coupling  these 
sensor/algorithm  systems  to  human  monitors  and  problem  holders.  To  meet  these  challenges  in  security  surveillance, 
we  have  developed  an  event-sensitivea  rchitecturew  here  machine  agents  provide  event-basedi  nformationt  o  human 
monitors  and  problem  holders  and  are  re-directable  given  contextual  information.  The  key  innovation  is  a  context- 
based  hierarchical  event  template  structure  which  can  be  used  to  integrate  data  over  a  distributed  sensor  system.a 


Woods,  D.  D.,  Patterson,  E.  S.,  &  Roth,  E.  M.  (2002).  Can  We  Ever  Escape  from  Data  Overload?  A  Cognitive 
Systems  Diagnosis.  Cognition,  Technology,  and  Work,  4(1):  pp.  22-36. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Agent;  Alarm;  Context;  Data  overload;  Information  Visualisation;  Workload 
Summary  abstract: 

Data  overload  is  a  generic  and  tremendously  difficult  problem  that  has  only  grown  with  each  new  wave  of 
technological  capabilities.  As  a  generic  and  persistent  problem,  three  observations  are  in  need  of  explanation:  Why 
is  data  overload  so  difficult  to  address?  Why  has  each  wave  of  technology  exacerbated,  rather  than  resolved,  data 
overload?  How  are  people,  as  adaptive  responsible  agents  in  context,  able  to  cope  with  the  challenge  of  data 
overload?  In  this  paper,  first  we  examine  three  different  characterisations  that  have  been  offered  to  capture  the 
nature  of  the  data  overload  problem  and  how  they  lead  to  different  proposed  solutions.  As  a  result,  we  propose  that 
(a)  data  overload  is  difficult  because  of  the  context  sensitivity  problem  -  meaning  lies,  not  in  data,  but  in 
relationships  of  data  to  interests  and  expectations  and  (b)  new  waves  of  technology  exacerbate  data  overload  when 
they  ignore  or  try  to  finesse  context  sensitivity.  The  paper  then  summarises  the  mechanisms  of  human  perception 
and  cognition  that  enable  people  to  focus  on  the  relevant  subset  of  the  available  data  despite  the  fact  that  what  is 
interesting  depends  on  context.  By  focusing  attention  on  the  root  issues  that  make  data  overload  a  difficult  problem 
and  on  people’s  fundamental  competence,  we  have  identified  a  set  of  constraints  that  all  potential  solutions  must 
meet.  Notable  among  these  constraints  is  the  idea  that  organisation  precedes  selectivity.  These  constraints  point 
toward  regions  of  the  solution  space  that  have  been  little  explored.  In  order  to  place  data  in  context,  designers  need 
to  display  data  in  a  conceptual  space  that  depicts  the  relationships,  events  and  contrasts  that  are  informative  in  a  field 
of  practice. 


Woods,  D.  D.  (2002).  Guts  or  No  Guts  (Grand  Unified  Theories):  Does/Can/Should  Cognitive  Engineering  Have 
G.U.T.S?  In  In  Proceedings  of  46th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Baltimore, 
MD. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

What  are  the  GUTs  of  Cognitive  Systems  Engineering  (CSE)?  G.U.T.  is  an  abbreviation  for  Grand  Unified  Theory. 
As  Cognitive  Science  matured,  Allen  Newell  proposed  a  unifying  model  of  cognition  expressed  as  a  software 
architecture  SOAR.  Similarly,  John  Anderson  developed  ACTR  also  claiming  it  represented  a  unified  theory  of 
cognition  in  the  form  of  a  computer  simulation.  Both  of  these  cognitive  architectures  are  computer  programs  that 
claim  to  simulate  or  be  the  basis  for  creating  simulations  of  how  people  perform  and  learn  cognitive  tasks.  Taking 
the  development  of  Cognitive  Science  as  a  possible  analogy  for  the  potential  development  of  Cognitive  Systems 
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Engineering,  this  panel  discussion  provides  a  platform  to  stimulate  a  vigorous  exchange  of  ideas  about  the 
foundation  of  and  potential  futures  of  CSE. 


Woods,  D.  D.,  &  Cook,  R.  I,  (2002).  Nine  Steps  to  Move  Forward  from  Error.  Cognition,  Technology,  and  Work, 
4(2),  137-144. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Error;  Failure  in  complex  systems;  Patient  safety 
Summary  abstract: 

Following  celebrated  failures  stakeholders  begin  to  ask  questions  about  how  to  improve  the  systems  and  processes 
they  operate,  manage  or  depend  on.  In  this  process  it  is  easy  to  become  stuck  on  the  label  ‘human  error’  as  if  it  were 
an  explanation  for  what  happened  and  as  if  such  a  diagnosis  specified  steps  to  improve.  To  guide  stakeholders  when 
celebrated  failure  or  other  developments  create  windows  of  opportunity  for  change  and  investment,  this  paper  draws 
on  generalizations  from  the  research  base  about  how  complex  systems  fail  and  about  how  people  contribute  to  safety 
and  risk  to  provide  a  set  of  Nine  Steps  forward  for  constructive  responses.  The  Nine  Steps  forward  are  described  and 
explained  in  the  form  of  series  of  maxims  and  corollaries  that  summarize  general  patterns  about  error  and  expertise, 
complexity  and  learning. 


Woods,  D.  D.  (2002).  Steering  the  Reverberations  of  Technology  Change  on  Fields  of  Practice:  Laws  That  Govern 
Cognitive  Work.  In  In  Proceedings  of  24th  Annual  Meeting  of  the  Cognitive  Science  Society.  [Plenary  Address]. 
Fairfax,  VA. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Research  on  cognitive  work  in  context  has  abstracted  a  set  of  common  patterns  about  cognitive  work  and  about  the 
relationship  of  people  and  computers.  I  offer  four  families  of  Laws  that  Govern  Cognitive  Work  plus  Norbert’s 
Contrast  as  a  synthesis  of  these  findings  to  guide  future  development  of  human-computer  cooperation.  These  Laws 
are  one  prong  of  a  general  strategy  to  avoid  repeats  of  past  "automation  surprises". 


Woods,  D.  D.  (2003).  Chapter  3.  Discovering  How  Distributed  Cognitive  Systems  Work.  In  E.  Hollnagel  (Ed.), 
Handbook  of  Cognitive  Task  Design  (pp.  37-53).  10  Industrial  Avenue,  Mahwah,  New  Jersey  07430:  Lawrence 
Erlbaum  Associates,  Inc. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

In  cognitive  work  there  is  a  mutual  adaptation  of  agents'  strategies,  affordances  of  artefacts,  and  demands  of  the 
work  setting.  Regularities  in  cognitive  work  concern  dynamic  interactions  across  these  sets  of  factors.  Because 
cognitive  work  systems  are  not  decomposable  into  independent  basic  elements,  different  empirical  tactics  are 
necessary,  though  core  values  of  observation,  discovery,  and  establishment  of  warrant  remain  fundamental  guides. 
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Woods,  D.  D.  (2003).  Using  Laws  That  Govern  Cognitive  Work  to  Design  Human-Automation  Systems.  In  In 
Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the 
Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S. 
Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures 
Summary  abstract: 

Research  on  cognitive  work  in  context  has  abstracted  a  set  of  common  patterns  about  cognitive  work  and  about  the 
relationship  of  people  and  computers.  I  offer  four  families  of  Laws  that  Govern  Cognitive  Work  plus  Norbert’s 
Contrast  as  a  synthesis  of  these  findings  to  guide  future  development  of  human-computer  cooperation.  These  Laws 
are  one  prong  of  a  general  strategy  to  avoid  repeats  of  past  “automation  surprises” 


Woods,  D.  D.  (2005).  Generic  Support  Requirements  for  Cognitive  Work:  Laws  That  Govern  Cognitive  Work  in 
Action.  In  In  Proceedings  of  49th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society.  Orlando,  FL. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

Research  in  Cognitive  Systems  Engineering  (CSE)  has  successfully  identified  basic  requirements  that  must  be  met  if 
new  technology  will  be  useful  to  practitioners  in  context.  Synthesizing  these  basic  requirements  or  support  functions 
is  part  of  a  process  of  debate  and  consolidation  of  the  foundations  of  the  field  after  25  years  of  productive  activity 
(Klein,  1999;  Endsley  et  al.,  2003;  Hollnagel  and  Woods,  2005).  This  work  takes  the  “Laws  that  Govern  Cognitive 
Work”  which  synthesize  basic  findings  aand  patterns  (Woods,  2002;  Hoffman  and  Woods,  2005)  and  provides  the 
next  step — a  set  of  basic  requirements  or  support  functions  for  design.  General  requirements  for  effective  support 
can  be  used  to  jump  start  individual  development  projects  in  any  domain.  Debating  how  to  achieve  these  support 
functions  helps  translate  the  insights  of  cognitive  work  analyses  into  tangible  new  uses  of  technological  possibilities. 


Woods,  D.  D.  (2005).  Plenary  Address:  Supporting  Cognitive  Work:  How  to  Achieve  High  Levels  of  Coordination 
and  Resilience  in  Joint  Cognitive  Systems  In  In  Proceedings  of  7th  International  Conference  on  Naturalistic 
Decision  Making.  Amsterdam,  The  Netherlands.  June  15-17,  2005. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

By  studying  joint  cognitive  systems  in  action,  field  oriented  research  has  done  more  than  identify  basic  patterns  in 
the  behavior— laws  that  govern  cognitive  work.  A  joint  cognitive  system  perspective  includes  the  artifacts  and  how 
they  shape  strategies  and  how  they  become  tools  through  use.  This  means  studying  joint  cognitive  systems  is  an 
examination  of  co-adaptive  cycles  as  people  in  different  roles  adapt  to  workaround  complexities  and  exploit  new 
capabilities.  Thus,  results  from  the  field  can  be  synthesized  as  a  set  of  basic  requirements  that  must  be  met  if  new 
technology  will  be  useful  to  practitioners  in  context.  Synthesizing  these  basic  requirements  or  support  functions  is  a 
critical  part  of  a  process  of  debate  and  consolidation  of  the  foundations  of  the  field  after  25  years  of  productive 
activity. 


Woods,  D.  D.  (2006).  Invited  Plenary  Address:  The  Law  of  Stretched  Systems  in  Action:  Exploiting  Robots.  In  In 
Proceedings  of  Human-Robot  Interaction  Conference  HRI'06,  Salt  Lake  City,  UT. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Human  Factors,  Design,  Measurement,  Reliability,  Automation  usage  decisions.  Trust,  Agent 
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Summary  abstract: 

Robotic  systems  represent  new  capabilities  that  justifiably  excite  technologists  and  problem  holders  in  many  areas. 
But  what  affordances  do  the  new  capabilities  represent  and  how  will  problem  holders  and  practitioners  exploit  these 
capabilities  as  they  struggle  to  meet  performance  demands  and  resource  pressures?  Discussions  of  the  impact  of  new 
robotic  technology  typically  mistake  new  capabilities  for  affordances  in  use.  The  dominant  note  is  that  robots  as 
autonomous  agents  will  revolutionize  human  activity.  This  is  a  fundamental  oversimplification  (see  Feltovich  et  al., 
2001)  as  past  research  has  shown  that  advances  in  autonomy  (an  intrinsic  capability)  have  turned  out  to  demand 
advances  in  support  for  coordinated  activity  (extrinsic  affordances). 

The  Law  of  Stretched  Systems  captures  the  co-adaptive  dynamic  that  human  leaders  under  pressure  for  higher  and 
more  efficient  levels  of  performance  will  exploit  new  capabilities  to  demand  more  complex  forms  of  work  (Woods 
and  Dekker,  2000;  Woods  and  Hollnagel,  2006).  This  law  provides  a  guide  to  use  past  findings  on  the  reverberations 
of  technology  change  to  project  how  effective  leaders  and  operators  will  exploit  the  capabilities  of  future  robotic 
systems.  When  one  applies  the  Law  of  Stretched  Systems  to  new  robotic  capabilities  for  demanding  work  settings, 
one  begins  to  see  new  stories  about  how  problem  holders  work  with  and  through  robotic  systems  to  accomplish 
goals.  These  are  not  stories  about  machine  autonomy  and  the  substitution  myth.  Rather,  the  new  capabilities  trigger 
the  exploration  of  new  story  lines  about  future  operations  that  concern: 

•  how  to  coordinate  activities  over,  wider  ranges, 

•  how  to  expand  our  perception  and  action  over  larger  spans  through  remote  devices,  and 

•  how  to  project  our  intent  into  distant  situations  to  achieve  our  goals. 

Research  on  these  story  lines  provide  new  results  on  awareness  of  remote  environments  through  robotic  systems  and 
brittleness/resilience  in  coordinating  people  and  robots  that  define  promising  directions  with  high  potential  return  for 
supporting  work  though  robotic  systems  (Woods  et  al.,  2004).  These  results  also  help  us  identify  new  candidates  for 
challenge  cases  in  HRI  and  new  classes  of  metrics  (e.g.,  the  fractal  path  scores  developed  by  Phillips  and  Voshel). 


Woods,  D.  D.,  Voshell,  M.,  Roesler,  A.,  Phillips,  F.,  Feil,  M.  and  Tittle,  J.  (2007).  Podcast-  the  Law  of  Stretched 
Systems  in  Action:  Exploiting  Robots.  In  Podcast  available  at  http://csel.eng.ohio-state.edu/podcasts/woods/. 

Key  Words: 

ADA  CTA;  Advanced  Decision  Architectures;  Collaborative  Technology  Alliance;  Conference  Proceedings 
Summary  abstract: 

David  Woods  delivers  the  opening  talk  for  the  first  annual  Human-Robot  Interaction  Conference  2006.  Robotic 
systems  represent  new  capabilities  that  justifiably  excite  technologists  and  problem  holders  in  many  areas.  But  what 
affordances  do  the  new  capabilities  represent  and  how  will  problem  holders  and  practitioners  exploit  these 
capabilities  as  they  struggle  to  meet  performance  demands  and  resource  pressures? 


Woods,  D.  D.  (2009).  Creating  Foresight:  Lessons  for  Enhancing  Resilience  from  Columbia.  In  B.  Starbuck  and  M. 
Farjoun  (Eds.),  Organization  at  the  Limit:  Nasa  and  the  Columbia  Disaster:  Blackwell,  (in  press). 

Key  Words: 
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Collaborative  Technology  Alliance 
Book  Chapter/Section 

Summary  abstract: 

The  past  seems  incredible,  the  future  implausible. 

To  look  forward  and  envision  how  organizations  can  achieve  very  high  reliability  and  resilience,  one  first  must  look 
back  with  clarity  unobscured  by  hindsight  bias.  The  Columbia  accident,  as  a  highly  visible  event  investigated  in 
depth  by  a  distinguished  and  independent  panel,  provides  an  opportunity  to  review  generic  patterns  seen  across 
multiple  accidents  and  across  studies  in  multiple  fields  of  practice  (Hollnagel,  1993).  This  chapter  examines  patterns 
present  in  the  Columbia  accident  (STS- 107)  in  order  to  consider  how  organizations  in  general  can  learn  and  change 
before  dramatic  failures  occur. 
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Woods,  D.  D.,  &  Hollnagel,  E.  (2006).  Joint  Cognitive  Systems:  Patterns  in  Cognitive  Systems  Engineering.:  CRC 
Press,  Taylor  &  Francis  Group,  6000  Broken  Sound  Parkway  NW,  Suite  300,  Boca  Raton,  FL  33487-2742. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Book 

CHAPTER  1:  CORE  ACTIVITIES  AND  VALUES 

Adaptability  versus  Limits 

Complementarity 

Core  Values  of  CSE  in  Practice 

On  Systems  in  CSE 

Patterns 

Discovering  Patterns  in  Joint  Cognitive  Systems  at  Work 
A  JCS  at  Work: 

CHAPTER  2:  JOINT  COGNITIVE  SYSTEMS  ADAPT  TO  COPE  WITH  COMPLEXITY 
Adaptation  in  Joint  Cognitive  Systems  at  Work 

CHAPTER  3:  BEING  BUMPABLE  (by  R.  I.  Cook) 

The  Story:  A  Delay 

The  Intensive  Care  Unit:  The  Scene,  the  Cast,  and  Backdrop 

Coping  with  Complexity:  Parceling  Out  Beds  by  the  Bedmeister 

Artifacts  as  Tools:  The  Bed  Book 

Preparing  for  Demand  >  Supply  Situations 

Son  of  Coping:  Building  an  ICU  from  Scratch 

Piling  Pelion  on  Ossa:  Escalating  Demands 

Observations  on  the  Incident 

CHAPTER  4:  DISCOVERY  AS  FUNCTIONAL  SYNTHESIS 
"Being  Bumpable"  as  an  Example  of  Studying  a  JCS  at  Work 
Insight  and  Functional  Synthesis 

CHAPTER  5:  SHAPING  THE  CONDITIONS  OF  OBSERVATION 
Three  Families  of  Methods 
Converging  Operations 
The  Psychologist's  Fallacy 

CHAPTER  6:  FUNCTIONAL  SYNTHESES,  LAWS,  AND  DESIGN 

Properties  of  Functional  Syntheses 

On  Laws  that  Govern  Joint  Cognitive  Systems  at  Work 

Challenges  to  Inform  Design 

Patterns  in  How  Joint  Cognitive  Systems  Work 

CHAPTER  7:  ARCHETYPICAL  STORIES  OF  JOINT  COGNITIVE  SYSTEMS  AT  WORK 
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The  gadget-minded  people  often  have  the  illusion  that  a  highly  automatized  world  will  make  smaller  claims  on 
human  ingenuity  than  does  the  present  one  ...  This  is  palpably  false. 

Norbert  Wiener,  1964,  p.  63 

For  almost  25  years,  Cognitive  Systems  Engineering  (CSE)  has  searched  out  and  listened  to  stories  of  claims  on 
human  ingenuity  as  fields  of  practice  have  changed  and  adapted  to  new  pressures,  new  devices,  and  new 
opportunities.  In  these  stories,  change  challenged  how  activities  are  coordinated,  how  systems  are  resilient  at 
boundaries,  and  how  artifacts  provide  affordances. 

This  meant  we  could  complement  the  previous  book  on  joint  cognitive  systems,  which  focused  on  the  foundational 
concepts,  with  a  book  that  used  stories  of  cycles  of  complexity  and  coping  to  show  the  main  values,  concepts  and 
approaches  of  CSE  in  action.  This  book  provides  a  way  to  look  at  our  collective  progress.  The  general  storylines 
make  up  a  base  of  [mdings  and  patterns  that  can  be  used  to  launch  new  studies  of  work,  to  project  the  effects  of  new 
rounds  of  change,  to  search  out  promising  directions  for  innovating  support.  Ironically,  thinking  about  representing  a 
research  base  as  a  set  of  abstract  narratives  is  in  itself  an  exploration  of  how  to  support  and  enhance  sharing 
technical,  scientific  and  design  information. 

The  stories  told  here  are  abo  ut  more  than  the  sharp  end  of  practice.  In  parallel,  each  story  tells  another  tale-revealing 
gaps  and  deficiencies  in  stories  we  tell  ourselves  about  our  relationship  with  technology,  about  how  we  accomplish 
our  goals,  and  how  our  systems  are  vulnerable  to  breakdown.  Acting  in  roles  as  researchers  or  designers  or 
managers  does  not  grant  us  any  immunity  from  the  processes  and  difficulties  of  coping  with  complexities. 

One  theme  running  through  this  volume  is-swprise-that  action  is  oriented  to  the  future,  anticipating  or  being 
prepared  for  what  comes  next.  This  book  is  intended  to  provide  a  start  on  a  pattern  base  to  help  us  be  prepared  to  be 
surprised: 

•  As  we  observe  joint  systems  at  work  in  cycles  of  change  and  adaptation  to  come, 

•  As  we  envision  with  others  how  joint  systems  will  work  given  new  pressures  and  opportunities,  and 

•  As  we  reflect  on  and  revise  our  beliefs  about  work  as  one  of  the  set  of  stakeholders  whose  beliefs  support  or 
hobble  joint  systems  as  they  work. 


Woods,  D.  D.,  &  Sarter,  N.  B.  (2010).  Capturing  the  Dynamics  of  Attention  Control  from  Individual  to  Distributed 
Systems:  The  Shape  of  Models  to  Come  Theoretical  Issues  in  Ergonomics  Science  (Special  Issue  on  Situation 
Awareness).  11(1-2),  7-28. 
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Summary  abstract: 

New  technology  presents  opportunities  for  enhancing  the  performance  of  human  systems  that  are  tasked  to  meet 
multiple,  often  competing  demands.  Yet,  mistakes  in  designing  and  deploying  these  technologies  can  create 
complexities  that  make  human  systems  more  brittle.  To  many  stakeholders,  the  answer  to  this  challenge  is  increase 
situation  awareness.  But  what  these  advocates  refer  to  when  they  talk  about  enhancing  situation  awareness  varies 
tremendously.  Over  15  years  ago,  the  authors  commented  on  how  the  label  was  ill  defined.  Today,  the  label  is  more 
popular  than  ever  but  the  range  of  situations  and  the  kinds  of  awareness  are  now  so  diverse  that  the  label  is  better 
referred  to  as  multiply  defined.  This  paper  returns  to  basic  concepts  and  findings  about  human  perception  and  the 
control  of  attention  and  the  critical  role  that  these  processes  play  in  individual  as  well  as  joint  and  distributed  activity 
-  how  people  know  where  to  focus  next  in  changing  situations.  This  paper  also  briefly  reviews  recent  studies  on  the 
neurobiology  of  the  control  of  attention  that  help  explain  how  people  find  what  is  relevant  despite  the  fact  that  this  is 
highly  context  sensitive.  Together,  the  findings  from  this  research  can  be  synthesised  into  new  models  that  capture 
how  human  systems  can  fluently  and  dynamically  shift  focus  as  context,  goals  and  situations  change.  These  models 
are  needed  to  be  able  to  understand,  predict  and  support  the  processes  involved  in  assessing  situations  and  achieving 
situation  awareness.  They  can  be  scaled  up  to  address  environments  where  technology  is  used  to  extend  human 
perception  into  distant  scenes  and  where  technology  connects  multiple  interdependent  agents  (both  human  groups 
and  machine  agents)  over  new  temporal  and  spatial  scales. 
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Woods,  D.  D.,  Tittle,  J.,  Feil,  M.,  &  Roesler,  A.  (2003).  Supporting  Human  and  Robot  Coordination.  In  In 
Proceedings  of  Collaborative  Technology  Alliances  Conference  2003.  Science  and  Technology  Innovations  for  the 
Objective  Force.  April  29  -  May  1,  2003.  University  of  Maryland  University  College,  Adelphi,  Maryland:  U.S. 
Army  Research  Laboratory. 

Key  Words: 

CTA,  Collaborative  Technology  Alliance,  Advanced  Decision  Architectures;  decision  making 
Summary  abstract: 

Developers  of  autonomous  capabilities  underestimate  the  need  for  coordination  when  their  automata  are  deployed 
into  complex  operational  settings.  Automata  are  brittle  as  literal  minded  agents  and  there  is  a  basic  asymmetry  in 
coordinative  competencies  between  people  and  automata.  The  new  capabilities  of  robotic  platforms  raise  new 
questions  about  how  to  support  coordination.  This  paper  presents  a  series  of  new  questions  for  innovation  in 
humanrobot  coordination.  These  include  supporting  people  in  their  roles  as  problem  holder  and  as  robotic  handler, 
overcoming  ambiguities  in  remote  perception,  avoiding  coordination  surprises  by  better  tools  to  see  into  future 
robotic  activities  and  contingencies,  and  responsibility  in  human-robot  teams. 


Woods,  D.  D.,  Tittle,  J.,  Feil,  M.,  &  Roesler,  A.  (2004).  Envisioning  Human-Robot  Coordination  for  Future 
Operations.  IEEE  SMC  Part  C,  34(2),  210-218. 
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Affordances,  design  methods,  human-automation  interaction,  human-robot-interaction,  presence,  remote  perception. 
Summary  abstract: 

Developers  of  autonomous  capabilities  underestimate  the  need  for  coordination  with  human  team  members  when 
their  automata  are  deployed  into  complex  operational  settings.  Automata  are  brittle  as  literal  minded  agents  and 
there  is  a  basic  asymmetry  in  coordinative  competencies  between  people  and  automata.  The  new  capabilities  of 
robotic  systems  raise  new  questions  about  how  to  support  coordination.  This  paper  presents  a  series  of  issues  that 
demand  innovation  to  achieve  human-robot  coordination  (HRC).  These  include  supporting  people  in  their  roles  as 
problem  holder  and  as  robotic  handler,  overcoming  ambiguities  in  remote  perception,  avoiding  coordination 
surprises  by  better  tools  to  see  into  future  robotic  activities  and  contingencies,  and  responsibility  in  human-robot 
teams. 


Yang,  G.-H.,  Jones,  L.  A.,  &  Kwon,  D.-S.  (2008).  Use  of  Simulated  Thermal  Cues  for  Material  Discrimination  and 
Identification  with  a  Multi-Fingered  Display.  Presence,  17(1),  29-42. 
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Summary  abstract: 

Thermal  cues  provide  information  about  the  thermal  properties  of  an  object  held  in  the  hand.  These  cues  can  be 
simulated  in  a  thermal  display  and  used  to  assist  in  identifying  the  object.  Two  experiments  were  conducted  using  a 
thermal  display  that  simulated  the  cues  associated  with  contact  with  different  materials.  The  thermal  contact  model 
was  based  on  a  semi-infinite  body  model  that  included  thermal  contact  resistance  and  blood  perfusion.  Its 
performance  was  evaluated  in  two  experiments,  the  first  of  which  involved  discriminating  between  simulated 
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materials,  and  in  the  second,  subjects  were  required  to  identify  simulated  materials  based  on  the  thermal  cues 
presented  to  one,  three,  or  five  fingers.  The  results  from  the  first  experiment  indicated  that  when  the  temperature 
profile  associated  with  contact  with  a  real  material  is  presented  to  the  finger,  subjects  can  use  this  cue  to 
discriminate  between  simulated  materials.  Their  performance  on  this  task  is  comparable  to  that  achieved  with  real 
materials  with  similar  thermal  properties.  In  the  second  experiment,  the  accuracy  with  which  subjects  identified  a 
simulated  material  based  on  thermal  cues  improved  as  the  number  of  fingers  stimulated  increased,  suggesting  that 
spatial  summation  of  cold  occurs  when  the  area  stimulated  is  noncontiguous.  However,  most  of  the  improvement  in 
identifying  materials  occurred  when  the  display  presented  thermal  cues  to  three  as  compared  to  one  finger,  with  little 
further  enhancement  in  performance  when  five  fingers  were  stimulated.  These  results  indicate  that  thermal  displays 
can  be  used  effectively  to  present  information  about  the  material  composition  of  objects  in  virtual  environments. 


Yang,  G.-H.,  Kwon,  D.-S.,  &  Jones,  L.  A.  (2009).  Spatial  Acuity  and  Summation  on  the  Hand:  The  Role  of  Thermal 
Cues  in  Material  Discrimination.  Attention,  Perception  &  Psychophysics.  The  Psychonomic  Society,  Inc.,  71(1), 
156-163. 

Key  Words: 

ADA  CTA 

Advanced  Decision  Architectures 
Collaborative  Technology  Alliance 
Journal  Article 

Summary  abstract: 

The  spatial  characteristics  of  thermal  perception  were  studied  in  two  experiments  that  examined  how  thermal  stimuli 
are  processed  within  the  hands.  A  thermal  display  that  simulates  cues  associated  with  making  contact  with  different 
materials  was  used  in  these  studies.  In  the  first  experiment,  participants  indicated  which  of  two  simulated  materials 
that  were  presented  to  the  index  fingertip  was  cooler.  The  results  indicated  that  participants  were  unable  to  resolve 
the  two  areas  of  thermal  stimulation.  In  the  second  experiment,  the  effects  of  concurrent  thermal  stimulation  on  the 
ability  to  discriminate  between  simulated  materials  were  evaluated.  Thermal  cues  were  presented  to  the  middle 
fingers  of  both  hands  and  to  two  adjacent  fingers  on  one  hand.  Thermal  spatial  summation  was  evident  across  the 
fingers,  which  enhanced  the  ability  to  discriminate  between  materials  when  the  cooler  stimulus  was  presented  to 
three  fingers.  When  the  same  stimulus  was  presented  to  the  two  hands,  the  stimulation  of  adjacent  fingers  altered  the 
perceived  thermal  response. 


Yen,  J.,  Strater,  L.,  McNeese,  M.,  Fan,  X.,  Cuevas,  H.  M.,  Oh,  S.,  Kim,  A.,  Minotra,  D.,  &  Hanratty,  T.  (2009). 
Chapter  12.  Cognitively-Inspired  Agents  as  Teammates  and  Decision  Aids.  In  P.  McDermott  and  L.  Allender 
(Eds.),  Advanced  Decision  Architectures  for  the  Warfighter:  Foundations  and  Technology.  Section  II.  Presenting 
Battlefield  Information  to  Warfighters:  48hiBooks  by  Alion  Science  &  Technology  Corporation  under  contract  to 
the  U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005,  Human  Research  and  Engineering 
Directorate.  Proceedings  from  the  Advanced  Decision  Architectures  (ADA)  group  of  the  Collaborative  Technology 
Aliance  (CTA)  Final  Research  Management  Board  Meeting  and  Workship  titled:  Supporting  the  Soldier  Decision 
Maker.  July  22-23,  2009.  1 100  New  Jersey  Avenue  SE,  Suite  200,  Washington,  D.C.  20003.  http://bldr- 
webtest. alionscience.com/AdaCta/ Agenda. aspx. 
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Summary  abstract: 

The  advancement  in  sensor,  communication,  and  information  technologies  have  resulted  in  information-dense 
environments  in  which  current  and  future  warfighters  must  operate.  In  the  face  of  dynamic,  constantly  changing 
events  and  conflicting  reports,  this  vast  amount  of  information  makes  it  difficult  for  warfighters  to  develop  and 
maintain  a  clear  picture  of  the  operational  situation.  This  difficulty  is  further  compounded  by  the  challenges  of 
asymmetric  warfare  where  complex  decisions  that  consider  multi-dimensional  factors  (e.g.,  human,  social,  and 
cultural  factors)  are  required  not  only  for  strategic  planning,  but  also  at  the  operational  and  tactical  level,  across  all 
echelons. 
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To  address  these  challenges,  this  research,  supported  by  the  Advanced  Decision  Architectures  Collaborative 
Technology  Alliance  of  the  U.S.  Army  Research  Lab,  aims  to  achieve  two  complimentary  research  goals.  Our  first 
research  objective  is  to  develop  a  cognitively- inspired  decision  agent  architecture  that  supports  the  delivery  of 
relevant  information  to  decision-makers  in  a  timely  fashion.  These  software  agents  act  as  both  teammates  and 
decision  aids,  enhance  the  human's  capability  in  reasoning  across  multiple  decision  spaces,  and  enable  dynamic 
team  collaboration  among  decision  makers.  With  its  cognitive  foundations,  functions  and  components  in  such  an 
architecture  can  be  better  related  to  the  processes  of  human  cognition.  Such  a  relationship  between  computational 
decision  aids  and  human  decision  process  can  contribute  to  the  co-refinement  of  both  the  computational  decision 
architecture  and  the  human-agent  team  cognition  theory. 

Our  second  research  objective  is  to  study  factors  that  affect  humanagent  interactions  such  that  warfighters  can  better 
calibrate  their  automation  usage  and  maintain  global  situation  awareness  (SA).  Despite  the  best  intentions  of 
automation  designers,  the  current  state  of  technology  does  not  produce  perfect  automation.  On  the  battlefield,  for 
example,  automation,  like  humans,  must  function  in  the  face  of  uncertainty  and  in  sub-optimal  environmental 
conditions.  Warfighters,  trained  to  operate  in  teams,  understand  that  these  complexities  may  affect  the  judgments  of 
their  human  teammates,  but  sometimes  fail  to  realize  that  agent  teammates  can  be  similarly  affected.  When 
warfighters  fail  to  adequately  understand  the  factors  influencing  the  performance  of  their  agent  teammates,  they  may 
make  poor  automation  usage  decisions  (AUDs)  and  may  fail  to  appropriately  trust  their  agent  teammates.  Previous 
research  has  shown  that  trust  in  team  members,  whether  human  or  agent,  is  critical  in  mediating  team  operations, 
particularly  with  increasing  levels  of  cognitive  complexity. 

The  objective  of  our  cognitively-inspired  agent  system  (R-CAST)  is  to  serve  as  decision  aids  and  teammates  of 
human  decision  makers.  There  are  many  normative,  descriptive,  and  prescriptive  decision  theories  that  can  be  built 
within  an  agent  system.  We  chose  to  adopt  Klein's  RPD  model  (Klein,  1997)  in  R-CAST  for  several  reasons.  First, 
the  RPD  model  is  a  holistic  model  of  human  decision  making  processes,  including  activities  such  as  seeking  relevant 
missing  information,  and  monitoring  expected  outcomes  of  decisions  to  detect  anomaly.  Second,  operators  can  more 
easily  understand  a  decision  aid  based  on  a  naturalistic  decision  process  (e.g.,  macro-cognitive  process  described  in 
RPD)  due  to  his/her  familiarity  and  experience  with  the  process.  This  encourages  active  human-agent  collaboration 
along  the  decision  process.  Such  a  decision  aid  would  be  intelligent  not  only  in  the  evaluation  and  choice  of  options, 
but  also  in  social  interactions  with  human  decision  makers. 

With  a  computational  RPD  model  as  the  macro-cognitive  process,  the  R-CAST  system  has  been  used  in  a  series  of 
experimental  studies  toward  a  better  understanding  of  human-agent  collaboration  in  time  stressed  decision  making 
situations.  The  first  experiment  was  conducted  to  help  us  to  understand  whether  future  warfighters  can  benefit  from 
R-CAST  in  handling  complex  multi-dimensional  decision  tasks.  Being  a  teammate  of  a  human  decision  maker,  the 
R-CAST  system  ought  to  be  tmstable  and  understandable.  The  second  and  third  experiments  examined  the  issue  of 
human-agent  trust  and  raised  the  interesting  question  of  how  to  facilitate  a  suitable  level  of  trust  between  warfighters 
and  decision  aids  to  Improve  the  overall  performance  of  the  human-agent  team.  The  fourth  experiment  was 
conducted  to  evaluate  the  impact  of  providing  a  mental  map  visualization  ofthe  agent's  decision  space  to  promote 
automation  transparency  on  effective  human  automation-usage  decisions. 

The  rest  of  the  chapter  is  organized  as  follows.  In  section  2,  we  describe  related  research  regarding  human-agent 
team  cognition  and  its  relationship  to  human  trust  in  agents.  In  section  3,  we  describe  the  RCAST  agent  architecture, 
which  is  empowered  by  a  computational  RPD  model.  In  section  4  we  give  an  overview  of  a  synthetic  task  designed 
for  studying  multi-dimensional  decision  making.  In  section  5,  we  summarize  a  series  of  human-in-the-Ioop 
experiments  where  the  R-CAST  agents  served  as  teammates  and  decision  aids,  investigating  the  issue  of 
multicontext  decision  making,  human-agent  trust,  and  mental  map  visualization.  Finally,  we  discuss  the  impacts  of 
our  work  and  point  to  some  directions  for  future  studies. 


Zachary,  W.,  Neville,  K.,  Fowlkes,  J.,  &  Hoffman,  R.  R.  (2007).  Human  Total  Cost  of  Ownership:  The  Penny 
Foolish  Principle  at  Work.  IEEE  Intelligent  Systems,  pp.  22-26. 
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This  essay  taps  a  recurring  topic  in  this  department:  procurement.  Human-centered  computing  is  about  creating 
technologies  that  are  intelligent  in  the  usual  sense  of  intelligent  systems.  But  we  also  mean  intelligent  in  the  sense 
that  they  are 

•  usable  versus  user-hostile; 

•  useful  because  they’re  designed  on  the  basis  of  results  from  cognitive  task  analysis,  so  that  they  actually  help 
people  do  things  that  need  to  be  done;  and 

•  understandable,  in  that  the  human  can  leam  what  the  machine  is  doing  and  why. 

Unfortunately,  the  procurement  processes  we  see  often  don’t  achieve  these  ends.  New  tools  called  decision  aids  or 
performance  aids  trigger  a  need  for  kluges — that  is,  they’re  rarely  usable.  Machines  wind  up  collecting  dust — 
they’re  not  useful.  Menus  cause  user  rage — they’re  not  understandable.  All  too  often  we  see  instances  of  this  Penny 
Foolish  Principle  of  complex  cognitive  systems: 

A  focus  on  short-term  cost  considerations,  as  a  main  driving  force  in  procuring  new  information  technology,  always 
comes  with  a  hefty  price  down  the  road,  a  price  that  always  weighs  most  heavily  on  users’  shoulders. 

Countless  program  managers,  system  designers,  computer  scientists,  human-factors  engineers,  and  cognitive 
systems  engineers  have  participated  in  projects  culminating  in  technologies  that  were  highly  constrained  by 
shortterm  cost  factors.  In  many  instances,  such  short-term  considerations  came  back  as  hauntings. 
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Summary  abstract: 

Across  information  analysis  domains,  it  is  often  difficult  to  recognize  when  analysis  is  inadequate  for  a  given 
context.  A  better  understanding  of  rigor  is  an  analytic  broadening  check  to  be  leveraged  against  this  uncertainty.  The 
purpose  of  this  research  is  to  refine  the  understanding  of  rigor,  exploring  the  concept  within  the  domain  of 
intelligence  analysis.  Nine  professional  intelligence  analysts  participated  in  a  study  of  how  analytic  rigor  is  judged. 
The  results  suggest  a  revised  definition  of  rigor,  reframing  it  as  an  emergent  multi-attribute  measure  of  sufficiency 
rather  than  as  a  measure  of  process  deviation.  Based  on  this  insight,  a  metric  for  assessing  rigor  was  developed, 
identifying  eight  attributes  of  rigorous  analysis.  Finally,  an  alternative  model  of  briefing  interactions  is  proposed  that 
integrates  this  framing  of  rigor  into  an  applied  context.  This  research,  although  specific  in  focus  to  intelligence 
analysis,  shows  the  potential  to  generalize  across  forms  of  information  analysis. 


Zelik,  D.  J.,  Patterson,  E.  S.,  &  Woods,  D.  D.  (2007).  Judging  Sufficiency:  How  Professional  Intelligence  Analysts 
Assess  Analytical  Rigor.  In  In  Proceedings  of  51st  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society. 
Baltimore,  MD. 
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Summary  abstract: 

This  study  examines  how  professional  intelligence  analysts  judge  the  rigor  behind  an  analysis.  The  study 
investigates  the  challenges  that  inhibit  the  understanding  of  rigor  in  intelligence  analysis  and  explores  cues  used  by 
analysts  to  identify  analytic  rigor — or  lack  of  rigor.  Nine  professional  intelligence  analysts  participated  in  a  modified 
elicitation  by  critiquing  method  study,  embedded  in  a  scenario  walkthrough.  Findings  from  the  study  indicate  that, 
while  professional  intelligence  analysts  can  make  perceptive  assessments  about  the  quality  of  an  analysis  process 
based  on  product  quality,  these  perceptions  are  apt  to  change  with  insight  into  the  analytic  process. 
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